[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:10:21 dist-test-slave-7034 krb5kdc[23761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:10:21 dist-test-slave-7034 krb5kdc[23761](info): set up 2 sockets
Sep 26 20:10:21 dist-test-slave-7034 krb5kdc[23761](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 20:10:22.320755 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:10:22 dist-test-slave-7034 krb5kdc[23761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:10:22.449802 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:39773
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:22.563239 23774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:22.563536 23774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:22.563580 23774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:22.567878 23774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:22.567946 23774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:22.567966 23774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:22.568014 23774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:22.568040 23774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:22.573508 23774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:39773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:39773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:22.574652 23774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:22.575721 23774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:10:22.585043 23774 server_base.cc:1034] running on GCE node
W20260926 20:10:22.586229 23779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:22.586427 23780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:22.586587 23782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:22.587208 23774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:22.588413 23774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:22.589582 23774 hybrid_clock.cc:648] HybridClock initialized: now 1790453422589556 us; error 41 us; skew 500 ppm
Sep 26 20:10:22 dist-test-slave-7034 krb5kdc[23761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453422, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:22.592506 23774 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:10:22.593421 23774 webserver.cc:460] Webserver started at http://127.23.50.190:39591/ using document root <none> and password file <none>
I20260926 20:10:22.594065 23774 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:22.594154 23774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:22.594391 23774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:22.596439 23774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "9eefce4eb0884f02a892c717284c1d59"
format_stamp: "Formatted at 2026-09-26 20:10:22 on dist-test-slave-7034"
server_key: "06548ab543bf3da9ab2d48a6559211fb"
server_key_iv: "eb7078262e21f38144861c66cc39d136"
server_key_version: "encryptionkey@0"
I20260926 20:10:22.597040 23774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "9eefce4eb0884f02a892c717284c1d59"
format_stamp: "Formatted at 2026-09-26 20:10:22 on dist-test-slave-7034"
server_key: "06548ab543bf3da9ab2d48a6559211fb"
server_key_iv: "eb7078262e21f38144861c66cc39d136"
server_key_version: "encryptionkey@0"
I20260926 20:10:22.600821 23774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:10:22.603293 23789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:22.604586 23774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:10:22.604733 23774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "9eefce4eb0884f02a892c717284c1d59"
format_stamp: "Formatted at 2026-09-26 20:10:22 on dist-test-slave-7034"
server_key: "06548ab543bf3da9ab2d48a6559211fb"
server_key_iv: "eb7078262e21f38144861c66cc39d136"
server_key_version: "encryptionkey@0"
I20260926 20:10:22.604859 23774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:22.623883 23774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:22.627230 23774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:22.627447 23774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:22.634749 23774 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:39773
I20260926 20:10:22.634780 23841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:39773 every 8 connection(s)
I20260926 20:10:22.635787 23774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:22.637046 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 23774
I20260926 20:10:22.637208 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:22.638923 23754 external_mini_cluster.cc:1428] Setting key 2c7ea09f699517838107628c7fb83bd1
I20260926 20:10:22.639263 23842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:22.646397 23842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59: Bootstrap starting.
I20260926 20:10:22.648918 23842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:22.649804 23842 log.cc:824] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:22.652091 23842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59: No bootstrap required, opened a new log
I20260926 20:10:22.657032 23842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eefce4eb0884f02a892c717284c1d59" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39773 } }
I20260926 20:10:22.657326 23842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:22.657426 23842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eefce4eb0884f02a892c717284c1d59, State: Initialized, Role: FOLLOWER
I20260926 20:10:22.658022 23842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eefce4eb0884f02a892c717284c1d59" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39773 } }
I20260926 20:10:22.658215 23842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:22.658299 23842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:22.658423 23842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:22.659797 23842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eefce4eb0884f02a892c717284c1d59" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39773 } }
Sep 26 20:10:22 dist-test-slave-7034 krb5kdc[23761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:22.660239 23842 leader_election.cc:302] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9eefce4eb0884f02a892c717284c1d59; no voters:
I20260926 20:10:22.660674 23842 leader_election.cc:288] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:22.660714 23847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:22.661068 23847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [term 1 LEADER]: Becoming Leader. State: Replica: 9eefce4eb0884f02a892c717284c1d59, State: Running, Role: LEADER
I20260926 20:10:22.661423 23847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eefce4eb0884f02a892c717284c1d59" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39773 } }
I20260926 20:10:22.662099 23842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:22.663218 23849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9eefce4eb0884f02a892c717284c1d59. Latest consensus state: current_term: 1 leader_uuid: "9eefce4eb0884f02a892c717284c1d59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eefce4eb0884f02a892c717284c1d59" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39773 } } }
I20260926 20:10:22.663350 23849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:22.663677 23848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9eefce4eb0884f02a892c717284c1d59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9eefce4eb0884f02a892c717284c1d59" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39773 } } }
I20260926 20:10:22.663811 23848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:22.663791 23856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:22.667603 23856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:22.668128 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:22.642690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.23.50.190:39773)
0926 20:10:22.643188 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:22.643207 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:22.644207 (+ 1000us) server_negotiation.cc:407] Connection header received
0926 20:10:22.648300 (+ 4093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:22.648317 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:22.648653 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:22.648950 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:22.651858 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:22.652919 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:22.655116 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:22.655594 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:22.655715 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:22.661126 (+ 5411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:22.661132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:22.661148 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:22.661190 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:22.663527 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:22.664205 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:22.664211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:22.664216 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:22.664383 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:22.664838 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:22.664842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:22.664844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:22.665375 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:22.665618 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:22.666463 (+ 845us) server_negotiation.cc:299] Negotiation successful
0926 20:10:22.666937 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":144,"threads_started":1}
I20260926 20:10:22.673699 23856 catalog_manager.cc:1347] Generated new cluster ID: ad957fa185464780b14e2ce87166b68e
I20260926 20:10:22.673806 23856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:22.683290 23856 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:22.684499 23856 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:22.693859 23856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9eefce4eb0884f02a892c717284c1d59: Generated new TSK 0
I20260926 20:10:22.694701 23856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:10:22.729804 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39773
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:22.844329 23868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:22.844597 23868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:22.844642 23868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:22.849115 23868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:22.849200 23868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:22.849264 23868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:10:22.854574 23868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:22.855688 23868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:22.856650 23868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:22.866261 23873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:22.866354 23874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:22.866291 23876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:22.866732 23868 server_base.cc:1034] running on GCE node
I20260926 20:10:22.867197 23868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:22.867858 23868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:22.869053 23868 hybrid_clock.cc:648] HybridClock initialized: now 1790453422869045 us; error 33 us; skew 500 ppm
Sep 26 20:10:22 dist-test-slave-7034 krb5kdc[23761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453422, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:22.872344 23868 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:10:22.873234 23868 webserver.cc:460] Webserver started at http://127.23.50.129:41823/ using document root <none> and password file <none>
I20260926 20:10:22.873767 23868 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:22.873826 23868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:22.874058 23868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:22.876050 23868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "160df8b67c134d209fd1ae435b3973ef"
format_stamp: "Formatted at 2026-09-26 20:10:22 on dist-test-slave-7034"
server_key: "9e0a85de74f2ff9806fb7af81c7db352"
server_key_iv: "ff6ba0d184abc1f93b996da71e8d2947"
server_key_version: "encryptionkey@0"
I20260926 20:10:22.876550 23868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "160df8b67c134d209fd1ae435b3973ef"
format_stamp: "Formatted at 2026-09-26 20:10:22 on dist-test-slave-7034"
server_key: "9e0a85de74f2ff9806fb7af81c7db352"
server_key_iv: "ff6ba0d184abc1f93b996da71e8d2947"
server_key_version: "encryptionkey@0"
I20260926 20:10:22.880442 23868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:10:22.882922 23883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:22.884155 23868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:10:22.884249 23868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "160df8b67c134d209fd1ae435b3973ef"
format_stamp: "Formatted at 2026-09-26 20:10:22 on dist-test-slave-7034"
server_key: "9e0a85de74f2ff9806fb7af81c7db352"
server_key_iv: "ff6ba0d184abc1f93b996da71e8d2947"
server_key_version: "encryptionkey@0"
I20260926 20:10:22.884338 23868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:22.898465 23868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:22.902343 23868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:22.902510 23868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:22.903051 23868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:22.903890 23868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:22.903937 23868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:22.903975 23868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:22.903991 23868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:22.913554 23868 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:35551
I20260926 20:10:22.913645 23996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:35551 every 8 connection(s)
I20260926 20:10:22.914563 23868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:10:22.916348 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 23868
I20260926 20:10:22.916471 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:10:22.916779 23754 external_mini_cluster.cc:1428] Setting key b420aff45ed8d5b22cd150d236579978
Sep 26 20:10:22 dist-test-slave-7034 krb5kdc[23761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453422, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
I20260926 20:10:22.931753 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:22.916528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:45239 (local address 127.23.50.190:39773)
0926 20:10:22.916689 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:22.916694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:22.917745 (+ 1051us) server_negotiation.cc:407] Connection header received
0926 20:10:22.919617 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:22.919619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:22.919668 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:22.919746 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:22.921231 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:22.921771 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:22.923926 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:22.924171 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:22.924256 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:22.927423 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:22.927430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:22.927433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:22.927458 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:22.929056 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:22.929606 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:22.929612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:22.929614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:22.929700 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:22.930109 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:22.930112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:22.930114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:22.930495 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:22.930747 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:22.931335 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 20:10:22.931493 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:10:22.932572 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:22.917011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39773 (local address 127.23.50.129:45239)
0926 20:10:22.917589 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:22.917650 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:22.919374 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:22.919965 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:22.919977 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:22.920492 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:22.921079 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:22.921095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:22.922017 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:22.922024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:22.923762 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:22.923778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:22.924362 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:22.924369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:22.924653 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:22.925368 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:22.925393 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:22.927225 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:22.929188 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:22.929194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:22.929212 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:22.929495 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:22.929816 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:22.929819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:22.929821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:22.930002 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:22.930755 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:22.930763 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:22.931036 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 20:10:22.931221 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 20:10:22.931564 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":175,"threads_started":1}
I20260926 20:10:22.934260 23997 heartbeater.cc:347] Connected to a master server at 127.23.50.190:39773
I20260926 20:10:22.934567 23997 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:22.935127 23997 heartbeater.cc:511] Master 127.23.50.190:39773 requested a full tablet report, sending...
I20260926 20:10:22.936830 23806 ts_manager.cc:194] Registered new tserver with Master: 160df8b67c134d209fd1ae435b3973ef (127.23.50.129:35551)
I20260926 20:10:22.938179 23806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:45239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:10:22.949721 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39773
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:23.064954 24002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:23.065251 24002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:23.065339 24002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:23.069483 24002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:23.069579 24002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:23.069677 24002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:10:23.075326 24002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:23.076400 24002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:23.077431 24002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:23.087033 24010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:23.087057 24008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:23.087013 24007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:23.087389 24002 server_base.cc:1034] running on GCE node
I20260926 20:10:23.087852 24002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:23.088570 24002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:23.089745 24002 hybrid_clock.cc:648] HybridClock initialized: now 1790453423089728 us; error 26 us; skew 500 ppm
Sep 26 20:10:23 dist-test-slave-7034 krb5kdc[23761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453423, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:23.093050 24002 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:10:23.094044 24002 webserver.cc:460] Webserver started at http://127.23.50.130:39803/ using document root <none> and password file <none>
I20260926 20:10:23.094587 24002 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:23.094647 24002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:23.094887 24002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:23.096868 24002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "3f41c52d90cd4dff88fd1e8ec2c651cb"
format_stamp: "Formatted at 2026-09-26 20:10:23 on dist-test-slave-7034"
server_key: "0dcff4915e08dfc18ef4ef1320c2404e"
server_key_iv: "8c0efc948250cb8ab28dcfa711e9ae31"
server_key_version: "encryptionkey@0"
I20260926 20:10:23.097410 24002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "3f41c52d90cd4dff88fd1e8ec2c651cb"
format_stamp: "Formatted at 2026-09-26 20:10:23 on dist-test-slave-7034"
server_key: "0dcff4915e08dfc18ef4ef1320c2404e"
server_key_iv: "8c0efc948250cb8ab28dcfa711e9ae31"
server_key_version: "encryptionkey@0"
I20260926 20:10:23.101168 24002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 20:10:23.103545 24017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:23.104710 24002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 20:10:23.104890 24002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "3f41c52d90cd4dff88fd1e8ec2c651cb"
format_stamp: "Formatted at 2026-09-26 20:10:23 on dist-test-slave-7034"
server_key: "0dcff4915e08dfc18ef4ef1320c2404e"
server_key_iv: "8c0efc948250cb8ab28dcfa711e9ae31"
server_key_version: "encryptionkey@0"
I20260926 20:10:23.105037 24002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:23.119297 24002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:23.122783 24002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:23.122988 24002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:23.123612 24002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:23.124450 24002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:23.124522 24002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:23.124596 24002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:23.124643 24002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:23.133882 24002 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:45865
I20260926 20:10:23.133900 24130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:45865 every 8 connection(s)
I20260926 20:10:23.134837 24002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:10:23.135938 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 24002
I20260926 20:10:23.136054 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:10:23.136345 23754 external_mini_cluster.cc:1428] Setting key 27e5debb7422f5eba4dec5390ae86a64
Sep 26 20:10:23 dist-test-slave-7034 krb5kdc[23761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453423, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
I20260926 20:10:23.150815 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:23.136956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:51225 (local address 127.23.50.190:39773)
0926 20:10:23.137107 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:23.137111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:23.138151 (+ 1040us) server_negotiation.cc:407] Connection header received
0926 20:10:23.139502 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:23.139505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:23.139565 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:23.139637 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:23.140945 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:23.141511 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:23.143481 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:23.143723 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:23.143817 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:23.146774 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:23.146777 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:23.146780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:23.146805 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:23.148414 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:23.149062 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:23.149066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:23.149068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:23.149151 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:23.149512 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:23.149515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:23.149517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:23.149719 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:23.149823 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:23.150498 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 20:10:23.150618 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:10:23.151882 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:23.137408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39773 (local address 127.23.50.130:51225)
0926 20:10:23.138004 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:23.138049 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:23.139260 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:23.139813 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:23.139820 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:23.140177 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:23.140772 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:23.140793 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:23.141648 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:23.141655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:23.143311 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:23.143330 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:23.143897 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:23.143905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:23.144142 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:23.144797 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:23.144822 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:23.146600 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:23.148578 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:23.148584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:23.148603 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:23.148951 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:23.149252 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:23.149256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:23.149258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:23.149409 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:23.149834 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:23.149865 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:23.150179 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 20:10:23.150423 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 20:10:23.150811 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":160,"threads_started":1}
I20260926 20:10:23.153396 24131 heartbeater.cc:347] Connected to a master server at 127.23.50.190:39773
I20260926 20:10:23.153684 24131 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:23.154305 24131 heartbeater.cc:511] Master 127.23.50.190:39773 requested a full tablet report, sending...
I20260926 20:10:23.155629 23806 ts_manager.cc:194] Registered new tserver with Master: 3f41c52d90cd4dff88fd1e8ec2c651cb (127.23.50.130:45865)
I20260926 20:10:23.156543 23806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:51225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:10:23.165556 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39773
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:23.280208 24136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:23.280467 24136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:23.280511 24136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:23.284811 24136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:23.284876 24136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:23.284937 24136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:10:23.290340 24136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39773
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:23.291389 24136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:23.292347 24136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:23.302182 24142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:23.302175 24141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:23.302175 24144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:23.302477 24136 server_base.cc:1034] running on GCE node
I20260926 20:10:23.303014 24136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:23.303629 24136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:23.304824 24136 hybrid_clock.cc:648] HybridClock initialized: now 1790453423304806 us; error 33 us; skew 500 ppm
Sep 26 20:10:23 dist-test-slave-7034 krb5kdc[23761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453423, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:23.307726 24136 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:10:23.308687 24136 webserver.cc:460] Webserver started at http://127.23.50.131:34141/ using document root <none> and password file <none>
I20260926 20:10:23.309327 24136 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:23.309391 24136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:23.309557 24136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:23.311599 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "65756c39f6df4ed6bea7974fa9ac9f3d"
format_stamp: "Formatted at 2026-09-26 20:10:23 on dist-test-slave-7034"
server_key: "19c805cbc1cd4aa8e6af7d800c5e12bd"
server_key_iv: "152eba05e0af48331f00b0230a373a7a"
server_key_version: "encryptionkey@0"
I20260926 20:10:23.312101 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "65756c39f6df4ed6bea7974fa9ac9f3d"
format_stamp: "Formatted at 2026-09-26 20:10:23 on dist-test-slave-7034"
server_key: "19c805cbc1cd4aa8e6af7d800c5e12bd"
server_key_iv: "152eba05e0af48331f00b0230a373a7a"
server_key_version: "encryptionkey@0"
I20260926 20:10:23.315774 24136 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 20:10:23.318094 24151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:23.319327 24136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:10:23.319420 24136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "65756c39f6df4ed6bea7974fa9ac9f3d"
format_stamp: "Formatted at 2026-09-26 20:10:23 on dist-test-slave-7034"
server_key: "19c805cbc1cd4aa8e6af7d800c5e12bd"
server_key_iv: "152eba05e0af48331f00b0230a373a7a"
server_key_version: "encryptionkey@0"
I20260926 20:10:23.319504 24136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:23.344503 24136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:23.348003 24136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:23.348191 24136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:23.348749 24136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:23.349576 24136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:23.349622 24136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:23.349663 24136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:23.349679 24136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:23.358788 24136 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:36129
I20260926 20:10:23.358819 24264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:36129 every 8 connection(s)
I20260926 20:10:23.359736 24136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:10:23.362373 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 24136
I20260926 20:10:23.362479 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:10:23.362766 23754 external_mini_cluster.cc:1428] Setting key 33e22fe1ebe76082cc8557aa26743897
Sep 26 20:10:23 dist-test-slave-7034 krb5kdc[23761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453423, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:23.373692 23844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:23.361655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:54515 (local address 127.23.50.190:39773)
0926 20:10:23.361797 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:23.361800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:23.362877 (+ 1077us) server_negotiation.cc:407] Connection header received
0926 20:10:23.364575 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:23.364578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:23.364628 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:23.364720 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:23.366045 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:23.366575 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:23.367699 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:23.367923 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:23.368002 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:23.370495 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:23.370498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:23.370500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:23.370522 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:23.371798 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:23.372282 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:23.372285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:23.372286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:23.372356 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:23.372686 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:23.372688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:23.372690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:23.372879 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:23.372962 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:23.373466 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 20:10:23.373560 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:10:23.374527 24267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:23.362068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39773 (local address 127.23.50.131:54515)
0926 20:10:23.362653 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:23.362703 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:23.364331 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:23.364871 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:23.364878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:23.365347 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:23.365902 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:23.365927 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:23.366699 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:23.366702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:23.367586 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:23.367595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:23.368050 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:23.368052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:23.368225 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:23.368868 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:23.368884 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:23.370352 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:23.371902 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:23.371907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:23.371918 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:23.372189 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:23.372427 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:23.372429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:23.372430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:23.372555 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:23.372979 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:23.372984 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:23.373248 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 20:10:23.373409 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 20:10:23.373647 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":152,"threads_started":1}
I20260926 20:10:23.375707 24265 heartbeater.cc:347] Connected to a master server at 127.23.50.190:39773
I20260926 20:10:23.375952 24265 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:23.376437 24265 heartbeater.cc:511] Master 127.23.50.190:39773 requested a full tablet report, sending...
I20260926 20:10:23.377581 23806 ts_manager.cc:194] Registered new tserver with Master: 65756c39f6df4ed6bea7974fa9ac9f3d (127.23.50.131:36129)
I20260926 20:10:23.378327 23806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:54515
I20260926 20:10:23.389091 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:10:23 dist-test-slave-7034 krb5kdc[23761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453423, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:10:23 dist-test-slave-7034 krb5kdc[23761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453423, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.50.129@KRBTEST.COM
I20260926 20:10:23.416639 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:23.406595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.23.50.129:35551)
0926 20:10:23.406956 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:23.406969 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:23.406994 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:10:23.407046 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:23.407051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:23.407250 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:23.407376 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:23.408084 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:23.409007 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:23.410029 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:23.410442 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:23.410535 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:23.412929 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:23.412939 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:23.412952 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:23.412980 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:23.414860 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:23.415260 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:23.415265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:23.415266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:23.415350 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:23.415620 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:23.415623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:23.415625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:23.415916 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:23.416099 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:23.416250 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 20:10:23.416418 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":136,"threads_started":1}
W20260926 20:10:23.417714 23911 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56894
Sep 26 20:10:23 dist-test-slave-7034 krb5kdc[23761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:10:23 dist-test-slave-7034 krb5kdc[23761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.129@KRBTEST.COM
I20260926 20:10:23.443262 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:23.435003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.23.50.129:35551)
0926 20:10:23.435169 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:23.435174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:23.435330 (+ 156us) server_negotiation.cc:407] Connection header received
0926 20:10:23.435500 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:23.435504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:23.435554 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:23.435638 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:23.436344 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:23.436861 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:23.437688 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:23.437988 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:23.438099 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:23.440384 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:23.440387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:23.440390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:23.440419 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:23.441595 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:23.442067 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:23.442071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:23.442073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:23.442154 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:23.442436 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:23.442441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:23.442442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:23.442607 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:23.442739 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:23.442970 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 20:10:23.443091 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:10:23.444329 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 23868
I20260926 20:10:23.450076 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 24002
I20260926 20:10:23.457787 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 24136
I20260926 20:10:23.463410 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 23774
2026-09-26T20:10:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2232 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:10:23 dist-test-slave-7034 krb5kdc[24279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:10:23 dist-test-slave-7034 krb5kdc[24279](info): set up 2 sockets
Sep 26 20:10:23 dist-test-slave-7034 krb5kdc[24279](info): commencing operation
krb5kdc: starting...
W20260926 20:10:24.538089 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:10:24 dist-test-slave-7034 krb5kdc[24279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:10:24.661417 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:45059
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:45059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:24.773737 24292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:24.774094 24292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:24.774163 24292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:24.778306 24292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:24.778389 24292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:24.778409 24292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:24.778426 24292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:24.778442 24292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:24.784564 24292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:45059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:45059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:24.785940 24292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:24.787076 24292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:24.796093 24298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:24.796185 24297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:24.796264 24292 server_base.cc:1034] running on GCE node
W20260926 20:10:24.796131 24300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:24.797055 24292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:24.798377 24292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:24.799608 24292 hybrid_clock.cc:648] HybridClock initialized: now 1790453424799573 us; error 56 us; skew 500 ppm
Sep 26 20:10:24 dist-test-slave-7034 krb5kdc[24279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453424, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:24.802584 24292 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:10:24.803510 24292 webserver.cc:460] Webserver started at http://127.23.50.190:38845/ using document root <none> and password file <none>
I20260926 20:10:24.804097 24292 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:24.804193 24292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:24.804459 24292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:24.806624 24292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "78533c80ec964d6d92a1a4dafb049f0e"
format_stamp: "Formatted at 2026-09-26 20:10:24 on dist-test-slave-7034"
server_key: "2d9aea4fec35e26ee04fd6f3bd56fab8"
server_key_iv: "c8d28cddc5cdd09dfd46526cec9f88f1"
server_key_version: "encryptionkey@0"
I20260926 20:10:24.807196 24292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "78533c80ec964d6d92a1a4dafb049f0e"
format_stamp: "Formatted at 2026-09-26 20:10:24 on dist-test-slave-7034"
server_key: "2d9aea4fec35e26ee04fd6f3bd56fab8"
server_key_iv: "c8d28cddc5cdd09dfd46526cec9f88f1"
server_key_version: "encryptionkey@0"
I20260926 20:10:24.810945 24292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:10:24.813416 24307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:24.814793 24292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:10:24.814937 24292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "78533c80ec964d6d92a1a4dafb049f0e"
format_stamp: "Formatted at 2026-09-26 20:10:24 on dist-test-slave-7034"
server_key: "2d9aea4fec35e26ee04fd6f3bd56fab8"
server_key_iv: "c8d28cddc5cdd09dfd46526cec9f88f1"
server_key_version: "encryptionkey@0"
I20260926 20:10:24.815055 24292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:24.838737 24292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:24.841969 24292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:24.842186 24292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:24.849362 24292 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:45059
I20260926 20:10:24.849416 24359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:45059 every 8 connection(s)
I20260926 20:10:24.850418 24292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:24.853327 24360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:24.858099 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 24292
I20260926 20:10:24.858222 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:24.858489 23754 external_mini_cluster.cc:1428] Setting key 07b0c065c61fc844ca65fcd9977cd092
I20260926 20:10:24.859373 24360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e: Bootstrap starting.
I20260926 20:10:24.861786 24360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:24.862763 24360 log.cc:824] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:24.865073 24360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e: No bootstrap required, opened a new log
Sep 26 20:10:24 dist-test-slave-7034 krb5kdc[24279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:24.869669 24360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78533c80ec964d6d92a1a4dafb049f0e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45059 } }
I20260926 20:10:24.869971 24360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:24.870052 24360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78533c80ec964d6d92a1a4dafb049f0e, State: Initialized, Role: FOLLOWER
I20260926 20:10:24.870543 24360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78533c80ec964d6d92a1a4dafb049f0e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45059 } }
I20260926 20:10:24.870692 24360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:24.870764 24360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:24.870860 24360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:24.871735 24360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78533c80ec964d6d92a1a4dafb049f0e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45059 } }
I20260926 20:10:24.872089 24360 leader_election.cc:302] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78533c80ec964d6d92a1a4dafb049f0e; no voters:
I20260926 20:10:24.872442 24360 leader_election.cc:288] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:24.872622 24365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:24.872823 24365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [term 1 LEADER]: Becoming Leader. State: Replica: 78533c80ec964d6d92a1a4dafb049f0e, State: Running, Role: LEADER
I20260926 20:10:24.873116 24365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78533c80ec964d6d92a1a4dafb049f0e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45059 } }
I20260926 20:10:24.873531 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:24.859888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.23.50.190:45059)
0926 20:10:24.860319 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:24.860347 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:24.860383 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:10:24.861028 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:24.861045 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:24.861387 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:24.861681 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:24.862618 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:24.863901 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:24.865041 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:24.865563 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:24.865682 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:24.868463 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:24.868469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:24.868483 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:24.868510 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:24.870413 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:24.870920 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:24.870925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:24.870930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:24.871082 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:24.871373 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:24.871377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:24.871379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:24.871774 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:24.871975 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:24.872262 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 20:10:24.872593 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":147,"threads_started":1}
I20260926 20:10:24.873741 24360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:24.875551 24366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78533c80ec964d6d92a1a4dafb049f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78533c80ec964d6d92a1a4dafb049f0e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45059 } } }
I20260926 20:10:24.875689 24366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:24.875957 24367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78533c80ec964d6d92a1a4dafb049f0e. Latest consensus state: current_term: 1 leader_uuid: "78533c80ec964d6d92a1a4dafb049f0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78533c80ec964d6d92a1a4dafb049f0e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45059 } } }
I20260926 20:10:24.876057 24367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:24.876755 24376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:24.879850 24376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:24.885262 24376 catalog_manager.cc:1347] Generated new cluster ID: ce4145a30bb54ebe825c2918c3259d97
I20260926 20:10:24.885337 24376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:24.921785 24376 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:24.922813 24376 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:24.931671 24376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78533c80ec964d6d92a1a4dafb049f0e: Generated new TSK 0
I20260926 20:10:24.932411 24376 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:10:24.965022 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45059
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:25.102850 24386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:25.103122 24386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:25.103168 24386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:25.107388 24386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:25.107450 24386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:25.107511 24386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:10:25.112876 24386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45059
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:25.113999 24386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:25.115032 24386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:25.124785 24392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:25.124912 24386 server_base.cc:1034] running on GCE node
W20260926 20:10:25.124809 24394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:25.124809 24391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:25.125502 24386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:25.126288 24386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:25.127478 24386 hybrid_clock.cc:648] HybridClock initialized: now 1790453425127459 us; error 32 us; skew 500 ppm
Sep 26 20:10:25 dist-test-slave-7034 krb5kdc[24279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453425, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:25.130625 24386 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:10:25.131625 24386 webserver.cc:460] Webserver started at http://127.23.50.129:42037/ using document root <none> and password file <none>
I20260926 20:10:25.132319 24386 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:25.132408 24386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:25.132599 24386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:25.135015 24386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "0cc29835ca8a432d9697ab81188ae58f"
format_stamp: "Formatted at 2026-09-26 20:10:25 on dist-test-slave-7034"
server_key: "05ef411732ebf015bfc4b5a14f641d35"
server_key_iv: "c1c710a47195b6ca844321de65993b02"
server_key_version: "encryptionkey@0"
I20260926 20:10:25.135636 24386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "0cc29835ca8a432d9697ab81188ae58f"
format_stamp: "Formatted at 2026-09-26 20:10:25 on dist-test-slave-7034"
server_key: "05ef411732ebf015bfc4b5a14f641d35"
server_key_iv: "c1c710a47195b6ca844321de65993b02"
server_key_version: "encryptionkey@0"
I20260926 20:10:25.139469 24386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:10:25.142056 24401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:25.143456 24386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:25.143599 24386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "0cc29835ca8a432d9697ab81188ae58f"
format_stamp: "Formatted at 2026-09-26 20:10:25 on dist-test-slave-7034"
server_key: "05ef411732ebf015bfc4b5a14f641d35"
server_key_iv: "c1c710a47195b6ca844321de65993b02"
server_key_version: "encryptionkey@0"
I20260926 20:10:25.143714 24386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:25.159277 24386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:25.162842 24386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:25.163079 24386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:25.163753 24386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:25.164680 24386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:25.164732 24386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:25.164810 24386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:25.164847 24386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:25.174746 24386 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:44135
I20260926 20:10:25.174773 24516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:44135 every 8 connection(s)
I20260926 20:10:25.175788 24386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:10:25.182624 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 24386
I20260926 20:10:25.182749 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:10:25.183024 23754 external_mini_cluster.cc:1428] Setting key 2fc56b3d18c1da3f95ee9f8b654e371f
Sep 26 20:10:25 dist-test-slave-7034 krb5kdc[24279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453425, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:25.189694 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.177745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:50803 (local address 127.23.50.190:45059)
0926 20:10:25.177953 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:25.177958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:25.178838 (+ 880us) server_negotiation.cc:407] Connection header received
0926 20:10:25.179686 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:25.179689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:25.179734 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:25.179821 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:25.180983 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.181505 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.182933 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.183220 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.183306 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.186161 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:25.186164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:25.186166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:25.186189 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:25.187659 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:25.188194 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:25.188202 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:25.188203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:25.188282 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:25.188614 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:25.188617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:25.188619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:25.188795 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:25.188895 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:25.189387 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 20:10:25.189524 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 20:10:25.190569 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.178210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:45059 (local address 127.23.50.129:50803)
0926 20:10:25.178674 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:25.178723 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:25.179476 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:25.179983 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:25.179990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:25.180378 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:25.180845 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.180858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.181652 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.181655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.182805 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.182815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.183373 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.183378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.183542 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.184282 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:25.184304 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:25.185997 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:25.187796 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:25.187802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:25.187817 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:25.188075 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:25.188370 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:25.188374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:25.188376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:25.188517 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:25.188895 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:25.188903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:25.189154 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 20:10:25.189334 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 20:10:25.189656 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":119,"threads_started":1}
I20260926 20:10:25.191792 24517 heartbeater.cc:347] Connected to a master server at 127.23.50.190:45059
I20260926 20:10:25.192044 24517 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:10:25.192606 24517 heartbeater.cc:511] Master 127.23.50.190:45059 requested a full tablet report, sending...
I20260926 20:10:25.194380 24319 ts_manager.cc:194] Registered new tserver with Master: 0cc29835ca8a432d9697ab81188ae58f (127.23.50.129:44135)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
I20260926 20:10:25.195622 24319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:50803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:10:25.213336 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45059
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:25.326354 24522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:25.326699 24522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:25.326747 24522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:25.330914 24522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:25.330968 24522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:25.331027 24522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:10:25.336277 24522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45059
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:25.337282 24522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:25.338390 24522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:10:25.348914 24522 server_base.cc:1034] running on GCE node
W20260926 20:10:25.348875 24528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:25.348874 24530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:25.348874 24527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:25.349618 24522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:25.350345 24522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:25.351529 24522 hybrid_clock.cc:648] HybridClock initialized: now 1790453425351493 us; error 53 us; skew 500 ppm
Sep 26 20:10:25 dist-test-slave-7034 krb5kdc[24279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453425, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:25.354526 24522 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:10:25.355438 24522 webserver.cc:460] Webserver started at http://127.23.50.130:33689/ using document root <none> and password file <none>
I20260926 20:10:25.356025 24522 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:25.356117 24522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:25.356354 24522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:25.358448 24522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "a9cdd15589fc49329791fabfee45f0c7"
format_stamp: "Formatted at 2026-09-26 20:10:25 on dist-test-slave-7034"
server_key: "471f849b908a7c792a6826e288634b30"
server_key_iv: "56b5e7efe540a91e106fbe32d508dab8"
server_key_version: "encryptionkey@0"
I20260926 20:10:25.358999 24522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "a9cdd15589fc49329791fabfee45f0c7"
format_stamp: "Formatted at 2026-09-26 20:10:25 on dist-test-slave-7034"
server_key: "471f849b908a7c792a6826e288634b30"
server_key_iv: "56b5e7efe540a91e106fbe32d508dab8"
server_key_version: "encryptionkey@0"
I20260926 20:10:25.362833 24522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:10:25.365237 24537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:25.366629 24522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:10:25.366775 24522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "a9cdd15589fc49329791fabfee45f0c7"
format_stamp: "Formatted at 2026-09-26 20:10:25 on dist-test-slave-7034"
server_key: "471f849b908a7c792a6826e288634b30"
server_key_iv: "56b5e7efe540a91e106fbe32d508dab8"
server_key_version: "encryptionkey@0"
I20260926 20:10:25.366900 24522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:25.389261 24522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:25.392283 24522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:25.392488 24522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:25.393093 24522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:25.393972 24522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:25.394038 24522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:25.394109 24522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:25.394160 24522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:25.403118 24522 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:33999
I20260926 20:10:25.403147 24651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:33999 every 8 connection(s)
I20260926 20:10:25.404017 24522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:10:25.410110 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 24522
I20260926 20:10:25.410231 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:10:25.410473 23754 external_mini_cluster.cc:1428] Setting key 6d35aeb1baa0565300420cc8a249611a
Sep 26 20:10:25 dist-test-slave-7034 krb5kdc[24279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453425, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:25.418016 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.405800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:55699 (local address 127.23.50.190:45059)
0926 20:10:25.405950 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:25.405954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:25.406865 (+ 911us) server_negotiation.cc:407] Connection header received
0926 20:10:25.407748 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:25.407751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:25.407799 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:25.407887 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:25.409072 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.409603 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.411006 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.411325 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.411454 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.414338 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:25.414341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:25.414346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:25.414376 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:25.415886 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:25.416422 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:25.416425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:25.416426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:25.416506 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:25.416850 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:25.416854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:25.416855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:25.417038 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:25.417143 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:25.417660 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 20:10:25.417779 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:10:25.418804 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.406240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:45059 (local address 127.23.50.130:55699)
0926 20:10:25.406691 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:25.406735 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:25.407539 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:25.408045 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:25.408052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:25.408423 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:25.408912 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.408926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.409747 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.409751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.410869 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.410880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.411497 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.411501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.411737 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.412426 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:25.412443 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:25.414163 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:25.416002 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:25.416009 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:25.416024 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:25.416302 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:25.416600 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:25.416604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:25.416606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:25.416746 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:25.417143 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:25.417151 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:25.417425 (+ 274us) client_negotiation.cc:769] Sending connection context
0926 20:10:25.417608 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 20:10:25.417968 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:10:25.419983 24652 heartbeater.cc:347] Connected to a master server at 127.23.50.190:45059
I20260926 20:10:25.420305 24652 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:25.420920 24652 heartbeater.cc:511] Master 127.23.50.190:45059 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
I20260926 20:10:25.422083 24319 ts_manager.cc:194] Registered new tserver with Master: a9cdd15589fc49329791fabfee45f0c7 (127.23.50.130:33999)
I20260926 20:10:25.422851 24319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:55699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:10:25.436779 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45059
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:25.548919 24657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:25.549197 24657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:25.549273 24657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:25.553507 24657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:25.553580 24657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:25.553642 24657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:10:25.559163 24657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45059
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:25.560290 24657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:25.561333 24657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:25.571098 24665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:25.571120 24662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:25.571098 24663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:25.571344 24657 server_base.cc:1034] running on GCE node
I20260926 20:10:25.571956 24657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:25.572610 24657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:25.573781 24657 hybrid_clock.cc:648] HybridClock initialized: now 1790453425573756 us; error 37 us; skew 500 ppm
Sep 26 20:10:25 dist-test-slave-7034 krb5kdc[24279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453425, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:25.576874 24657 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:10:25.577898 24657 webserver.cc:460] Webserver started at http://127.23.50.131:32801/ using document root <none> and password file <none>
I20260926 20:10:25.578503 24657 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:25.578598 24657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:25.578837 24657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:25.580941 24657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "2db59f368a8a4367b2f8bde0ed8711cd"
format_stamp: "Formatted at 2026-09-26 20:10:25 on dist-test-slave-7034"
server_key: "068b76a04102c7237123822a347f5e0c"
server_key_iv: "01f4b07c25974d6c84c62b51fcc75999"
server_key_version: "encryptionkey@0"
I20260926 20:10:25.581573 24657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "2db59f368a8a4367b2f8bde0ed8711cd"
format_stamp: "Formatted at 2026-09-26 20:10:25 on dist-test-slave-7034"
server_key: "068b76a04102c7237123822a347f5e0c"
server_key_iv: "01f4b07c25974d6c84c62b51fcc75999"
server_key_version: "encryptionkey@0"
I20260926 20:10:25.585494 24657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 20:10:25.588358 24672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:25.589747 24657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:10:25.589927 24657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "2db59f368a8a4367b2f8bde0ed8711cd"
format_stamp: "Formatted at 2026-09-26 20:10:25 on dist-test-slave-7034"
server_key: "068b76a04102c7237123822a347f5e0c"
server_key_iv: "01f4b07c25974d6c84c62b51fcc75999"
server_key_version: "encryptionkey@0"
I20260926 20:10:25.590065 24657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:25.606369 24657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:25.609819 24657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:25.610117 24657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:25.611035 24657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:25.612037 24657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:25.612110 24657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:25.612179 24657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:25.612210 24657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:25.621675 24657 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:38631
I20260926 20:10:25.621721 24786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:38631 every 8 connection(s)
I20260926 20:10:25.622751 24657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:10:25 dist-test-slave-7034 krb5kdc[24279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453425, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:25.632876 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 24657
I20260926 20:10:25.632997 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:10:25.633271 23754 external_mini_cluster.cc:1428] Setting key 2ca15c8a6b28ed095b09a8001e557426
I20260926 20:10:25.636827 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.624643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:34915 (local address 127.23.50.190:45059)
0926 20:10:25.624782 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:25.624786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:25.625969 (+ 1183us) server_negotiation.cc:407] Connection header received
0926 20:10:25.626924 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:25.626927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:25.626983 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:25.627083 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:25.628318 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.628865 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.630086 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.630315 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.630427 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.632891 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:25.632896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:25.632904 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:25.632950 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:25.634720 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:25.635314 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:25.635319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:25.635320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:25.635409 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:25.635826 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:25.635829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:25.635831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:25.636027 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:25.636115 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:25.636603 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 20:10:25.636706 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:10:25.637643 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.625065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:45059 (local address 127.23.50.131:34915)
0926 20:10:25.625765 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:25.625807 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:25.626688 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:25.627309 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:25.627315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:25.627671 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:25.628166 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.628180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.629009 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.629012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.629966 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.629975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.630481 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.630485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.630662 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.631291 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:25.631308 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:25.632724 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:25.634847 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:25.634852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:25.634866 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:25.635166 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:25.635524 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:25.635529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:25.635531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:25.635679 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:25.636123 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:25.636128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:25.636366 (+ 238us) client_negotiation.cc:769] Sending connection context
0926 20:10:25.636525 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 20:10:25.636780 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":117,"threads_started":1}
I20260926 20:10:25.638789 24787 heartbeater.cc:347] Connected to a master server at 127.23.50.190:45059
I20260926 20:10:25.639050 24787 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:25.639552 24787 heartbeater.cc:511] Master 127.23.50.190:45059 requested a full tablet report, sending...
I20260926 20:10:25.640532 24319 ts_manager.cc:194] Registered new tserver with Master: 2db59f368a8a4367b2f8bde0ed8711cd (127.23.50.131:38631)
I20260926 20:10:25.641114 24319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:34915
I20260926 20:10:25.647145 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:10:25.658558 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.650748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.23.50.190:45059)
0926 20:10:25.650910 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:25.650915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:25.651074 (+ 159us) server_negotiation.cc:407] Connection header received
0926 20:10:25.651217 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:25.651220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:25.651277 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:25.651364 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:25.652063 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.652572 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.653405 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.653622 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.653753 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.655305 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:25.655309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:25.655311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:25.655343 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:25.656658 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:25.657085 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:25.657090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:25.657092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:25.657175 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:25.657449 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:25.657453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:25.657455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:25.657684 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:25.657854 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:25.658189 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 20:10:25.658370 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:10:25.663335 24319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:10:25.666343 24319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:25.681917 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.674954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:38631 (local address 127.0.0.1:56168)
0926 20:10:25.675402 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:25.675452 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:25.675582 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:25.676795 (+ 1213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:25.676801 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:25.676826 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:25.676978 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.676988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.679055 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.679058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.680189 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.680200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.681409 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.681414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.681526 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.681568 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:10:25.681626 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:10:25.681743 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":219,"thread_start_us":74,"threads_started":1}
I20260926 20:10:25.681927 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.675926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.23.50.129:44135)
0926 20:10:25.676198 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:25.676213 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:25.677162 (+ 949us) server_negotiation.cc:407] Connection header received
0926 20:10:25.677277 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:25.677289 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:25.677488 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:25.677628 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:25.678000 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.678743 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.680627 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.681189 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.681327 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.681492 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:25.681621 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 20:10:25.681729 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":115,"threads_started":1}
I20260926 20:10:25.682360 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.675145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:33999 (local address 127.0.0.1:34204)
0926 20:10:25.676648 (+ 1503us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:25.676662 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:25.676802 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:25.677392 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:25.677395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:25.677411 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:25.677495 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.677502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.678953 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.678957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.680472 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.680484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.681916 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.681919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.682028 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.682053 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:10:25.682135 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:10:25.682210 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1420,"mutex_wait_us":88,"thread_start_us":104,"threads_started":1}
I20260926 20:10:25.682385 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.675118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56168 (local address 127.23.50.131:38631)
0926 20:10:25.676256 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:25.676277 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:25.676302 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:10:25.676383 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:25.676389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:25.676614 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:25.676823 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:25.677092 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.678482 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.680336 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.681245 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.681739 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.682070 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:25.682130 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 20:10:25.682241 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":212,"threads_started":1}
I20260926 20:10:25.682896 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.676230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.23.50.130:33999)
0926 20:10:25.676658 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:25.676674 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:25.676935 (+ 261us) server_negotiation.cc:407] Connection header received
0926 20:10:25.677024 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:25.677029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:25.677244 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:25.677417 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:25.677592 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.678782 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.680615 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.681509 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.681631 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.681803 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:25.682633 (+ 830us) server_negotiation.cc:299] Negotiation successful
0926 20:10:25.682751 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":102,"threads_started":1}
I20260926 20:10:25.683212 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.675779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:44135 (local address 127.0.0.1:48466)
0926 20:10:25.677070 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:25.677085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:25.677170 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:25.677758 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:25.677760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:25.677780 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:25.677837 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.677884 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.678877 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.678882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.680475 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.680484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.681327 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.681331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.681440 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.681477 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:10:25.681592 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:10:25.682864 (+ 1272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"mutex_wait_us":241,"thread_start_us":121,"threads_started":1}
I20260926 20:10:25.685491 24585 tablet_service.cc:1467] Processing CreateTablet for tablet c68f72b2b15c4fa8ad2e4ea41d5ec926 (DEFAULT_TABLE table=test-table [id=70aa53d1f49f450fbf1d2333072ef53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:25.685709 24720 tablet_service.cc:1467] Processing CreateTablet for tablet c68f72b2b15c4fa8ad2e4ea41d5ec926 (DEFAULT_TABLE table=test-table [id=70aa53d1f49f450fbf1d2333072ef53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:25.685904 24450 tablet_service.cc:1467] Processing CreateTablet for tablet c68f72b2b15c4fa8ad2e4ea41d5ec926 (DEFAULT_TABLE table=test-table [id=70aa53d1f49f450fbf1d2333072ef53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:25.686671 24585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68f72b2b15c4fa8ad2e4ea41d5ec926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:25.686870 24450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68f72b2b15c4fa8ad2e4ea41d5ec926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:25.686875 24720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68f72b2b15c4fa8ad2e4ea41d5ec926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:25.692862 24804 tablet_bootstrap.cc:492] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7: Bootstrap starting.
I20260926 20:10:25.694579 24805 tablet_bootstrap.cc:492] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f: Bootstrap starting.
I20260926 20:10:25.695263 24806 tablet_bootstrap.cc:492] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd: Bootstrap starting.
I20260926 20:10:25.696852 24804 tablet_bootstrap.cc:654] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:25.696897 24805 tablet_bootstrap.cc:654] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:25.697198 24806 tablet_bootstrap.cc:654] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:25.697750 24805 log.cc:824] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:25.697750 24804 log.cc:824] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:25.698503 24806 log.cc:824] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:25.699851 24805 tablet_bootstrap.cc:492] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f: No bootstrap required, opened a new log
I20260926 20:10:25.699882 24804 tablet_bootstrap.cc:492] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7: No bootstrap required, opened a new log
I20260926 20:10:25.700048 24805 ts_tablet_manager.cc:1397] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:10:25.700047 24804 ts_tablet_manager.cc:1397] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260926 20:10:25.700435 24806 tablet_bootstrap.cc:492] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd: No bootstrap required, opened a new log
I20260926 20:10:25.700649 24806 ts_tablet_manager.cc:1397] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:10:25.704046 24806 raft_consensus.cc:348] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38631 } } peers { permanent_uuid: "a9cdd15589fc49329791fabfee45f0c7" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33999 } } peers { permanent_uuid: "0cc29835ca8a432d9697ab81188ae58f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44135 } }
I20260926 20:10:25.704327 24806 raft_consensus.cc:374] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:25.704386 24806 raft_consensus.cc:729] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2db59f368a8a4367b2f8bde0ed8711cd, State: Initialized, Role: FOLLOWER
I20260926 20:10:25.704247 24804 raft_consensus.cc:348] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38631 } } peers { permanent_uuid: "a9cdd15589fc49329791fabfee45f0c7" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33999 } } peers { permanent_uuid: "0cc29835ca8a432d9697ab81188ae58f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44135 } }
I20260926 20:10:25.704527 24804 raft_consensus.cc:374] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:25.704394 24805 raft_consensus.cc:348] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38631 } } peers { permanent_uuid: "a9cdd15589fc49329791fabfee45f0c7" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33999 } } peers { permanent_uuid: "0cc29835ca8a432d9697ab81188ae58f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44135 } }
I20260926 20:10:25.704609 24805 raft_consensus.cc:374] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:25.704598 24804 raft_consensus.cc:729] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9cdd15589fc49329791fabfee45f0c7, State: Initialized, Role: FOLLOWER
I20260926 20:10:25.704670 24805 raft_consensus.cc:729] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc29835ca8a432d9697ab81188ae58f, State: Initialized, Role: FOLLOWER
I20260926 20:10:25.704936 24806 consensus_queue.cc:260] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38631 } } peers { permanent_uuid: "a9cdd15589fc49329791fabfee45f0c7" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33999 } } peers { permanent_uuid: "0cc29835ca8a432d9697ab81188ae58f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44135 } }
I20260926 20:10:25.705101 24804 consensus_queue.cc:260] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38631 } } peers { permanent_uuid: "a9cdd15589fc49329791fabfee45f0c7" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33999 } } peers { permanent_uuid: "0cc29835ca8a432d9697ab81188ae58f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44135 } }
I20260926 20:10:25.705101 24805 consensus_queue.cc:260] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38631 } } peers { permanent_uuid: "a9cdd15589fc49329791fabfee45f0c7" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33999 } } peers { permanent_uuid: "0cc29835ca8a432d9697ab81188ae58f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44135 } }
I20260926 20:10:25.705788 24652 heartbeater.cc:503] Master 127.23.50.190:45059 was elected leader, sending a full tablet report...
I20260926 20:10:25.706118 24806 ts_tablet_manager.cc:1428] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260926 20:10:25.706202 24804 ts_tablet_manager.cc:1428] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260926 20:10:25.706265 24805 ts_tablet_manager.cc:1428] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:10:25.706454 24787 heartbeater.cc:503] Master 127.23.50.190:45059 was elected leader, sending a full tablet report...
I20260926 20:10:25.706681 24517 heartbeater.cc:503] Master 127.23.50.190:45059 was elected leader, sending a full tablet report...
W20260926 20:10:25.873797 24788 tablet.cc:2369] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:10:25.905215 24653 tablet.cc:2369] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:10:25.927291 24518 tablet.cc:2369] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:10:25.991794 24810 raft_consensus.cc:482] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:25.992075 24810 raft_consensus.cc:504] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38631 } } peers { permanent_uuid: "a9cdd15589fc49329791fabfee45f0c7" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33999 } } peers { permanent_uuid: "0cc29835ca8a432d9697ab81188ae58f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44135 } }
I20260926 20:10:25.993372 24810 leader_election.cc:288] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9cdd15589fc49329791fabfee45f0c7 (127.23.50.130:33999), 0cc29835ca8a432d9697ab81188ae58f (127.23.50.129:44135)
I20260926 20:10:25.997948 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.993619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:33999 (local address 127.23.50.131:46989)
0926 20:10:25.993757 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:25.993777 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:25.993914 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:25.994349 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:25.994352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:25.994378 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:25.994482 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.994487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.995504 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.995507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.996481 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.996490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.997439 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.997443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.997564 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.997591 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:10:25.997696 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 20:10:25.997771 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 20:10:25.997982 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.993745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:46989 (local address 127.23.50.130:33999)
0926 20:10:25.993924 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:25.993932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:25.993955 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:10:25.994053 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:25.994057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:25.994117 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:25.994220 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:25.994609 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.995389 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.996624 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.997300 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.997410 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.997454 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:25.997751 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 20:10:25.997822 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:10:25.998538 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.994107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:50345 (local address 127.23.50.129:44135)
0926 20:10:25.994261 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:25.994269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:25.994618 (+ 349us) server_negotiation.cc:407] Connection header received
0926 20:10:25.994778 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:25.994781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:25.994827 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:25.994900 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:25.995221 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.995876 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.997296 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.997938 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.998062 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.998118 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:25.998369 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 20:10:25.998448 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:10:25.998551 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:25.994104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:44135 (local address 127.23.50.131:50345)
0926 20:10:25.994490 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:25.994502 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:25.994632 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:25.994947 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:25.994950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:25.994970 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:25.995093 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.995099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.996002 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.996005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.997153 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:25.997164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:25.998088 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:25.998092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:25.998199 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:25.998223 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:10:25.998347 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 20:10:25.998426 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":158,"threads_started":1}
I20260926 20:10:25.998689 24606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68f72b2b15c4fa8ad2e4ea41d5ec926" candidate_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9cdd15589fc49329791fabfee45f0c7" is_pre_election: true
I20260926 20:10:25.999014 24606 raft_consensus.cc:2454] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db59f368a8a4367b2f8bde0ed8711cd in term 0.
I20260926 20:10:25.999063 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68f72b2b15c4fa8ad2e4ea41d5ec926" candidate_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cc29835ca8a432d9697ab81188ae58f" is_pre_election: true
I20260926 20:10:25.999346 24470 raft_consensus.cc:2454] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2db59f368a8a4367b2f8bde0ed8711cd in term 0.
I20260926 20:10:25.999559 24674 leader_election.cc:302] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2db59f368a8a4367b2f8bde0ed8711cd, a9cdd15589fc49329791fabfee45f0c7; no voters:
I20260926 20:10:25.999868 24810 raft_consensus.cc:2781] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:25.999967 24810 raft_consensus.cc:482] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:26.000028 24810 raft_consensus.cc:3037] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:26.001091 24810 raft_consensus.cc:504] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38631 } } peers { permanent_uuid: "a9cdd15589fc49329791fabfee45f0c7" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33999 } } peers { permanent_uuid: "0cc29835ca8a432d9697ab81188ae58f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44135 } }
I20260926 20:10:26.001439 24810 leader_election.cc:288] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [CANDIDATE]: Term 1 election: Requested vote from peers a9cdd15589fc49329791fabfee45f0c7 (127.23.50.130:33999), 0cc29835ca8a432d9697ab81188ae58f (127.23.50.129:44135)
I20260926 20:10:26.001778 24606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68f72b2b15c4fa8ad2e4ea41d5ec926" candidate_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9cdd15589fc49329791fabfee45f0c7"
I20260926 20:10:26.001899 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68f72b2b15c4fa8ad2e4ea41d5ec926" candidate_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cc29835ca8a432d9697ab81188ae58f"
I20260926 20:10:26.001926 24606 raft_consensus.cc:3037] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:26.002010 24470 raft_consensus.cc:3037] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:26.003232 24606 raft_consensus.cc:2454] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P a9cdd15589fc49329791fabfee45f0c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db59f368a8a4367b2f8bde0ed8711cd in term 1.
I20260926 20:10:26.003227 24470 raft_consensus.cc:2454] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 0cc29835ca8a432d9697ab81188ae58f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2db59f368a8a4367b2f8bde0ed8711cd in term 1.
I20260926 20:10:26.003584 24674 leader_election.cc:302] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cc29835ca8a432d9697ab81188ae58f, 2db59f368a8a4367b2f8bde0ed8711cd; no voters:
I20260926 20:10:26.003784 24810 raft_consensus.cc:2781] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:26.004050 24810 raft_consensus.cc:686] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [term 1 LEADER]: Becoming Leader. State: Replica: 2db59f368a8a4367b2f8bde0ed8711cd, State: Running, Role: LEADER
I20260926 20:10:26.004405 24810 consensus_queue.cc:237] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38631 } } peers { permanent_uuid: "a9cdd15589fc49329791fabfee45f0c7" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33999 } } peers { permanent_uuid: "0cc29835ca8a432d9697ab81188ae58f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44135 } }
I20260926 20:10:26.007602 24323 catalog_manager.cc:5495] T c68f72b2b15c4fa8ad2e4ea41d5ec926 P 2db59f368a8a4367b2f8bde0ed8711cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2db59f368a8a4367b2f8bde0ed8711cd (127.23.50.131). New cstate: current_term: 1 leader_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2db59f368a8a4367b2f8bde0ed8711cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9cdd15589fc49329791fabfee45f0c7" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cc29835ca8a432d9697ab81188ae58f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44135 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:10:26 dist-test-slave-7034 krb5kdc[24279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.129@KRBTEST.COM
I20260926 20:10:26.053213 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:26.043527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48476 (local address 127.23.50.129:44135)
0926 20:10:26.043723 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:26.043727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:26.043796 (+ 69us) server_negotiation.cc:407] Connection header received
0926 20:10:26.043908 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:26.043910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:26.043958 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:26.044028 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:26.044825 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:26.045354 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:26.046372 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:26.046590 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:26.046681 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:26.049553 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:26.049566 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:26.049582 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:26.049608 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:26.051429 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:26.051883 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:26.051889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:26.051890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:26.051970 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:26.052254 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:26.052258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:26.052259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:26.052508 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:26.052729 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:26.052902 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 20:10:26.053024 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 20:10:26 dist-test-slave-7034 krb5kdc[24279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453426, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:10:26 dist-test-slave-7034 krb5kdc[24279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453426, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.50.129@KRBTEST.COM
I20260926 20:10:26.079869 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:26.071165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.23.50.129:44135)
0926 20:10:26.071428 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:26.071431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:26.071450 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:10:26.071585 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:26.071587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:26.071629 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:26.071697 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:26.072352 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:26.072863 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:26.073726 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:26.073995 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:26.074080 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:26.076539 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:26.076542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:26.076544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:26.076568 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:26.078254 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:26.078661 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:26.078663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:26.078665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:26.078761 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:26.079011 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:26.079013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:26.079014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:26.079167 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:26.079289 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:26.079542 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 20:10:26.079642 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
Sep 26 20:10:26 dist-test-slave-7034 krb5kdc[24279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:10:26 dist-test-slave-7034 krb5kdc[24279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.129@KRBTEST.COM
I20260926 20:10:26.105212 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:26.096918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48500 (local address 127.23.50.129:44135)
0926 20:10:26.097110 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:26.097116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:26.097254 (+ 138us) server_negotiation.cc:407] Connection header received
0926 20:10:26.097443 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:26.097447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:26.097491 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:26.097576 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:26.098355 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:26.098874 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:26.099738 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:26.099960 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:26.100069 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:26.102437 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:26.102442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:26.102447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:26.102475 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:26.103645 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:26.104052 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:26.104056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:26.104057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:26.104132 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:26.104406 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:26.104409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:26.104411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:26.104568 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:26.104701 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:26.104929 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 20:10:26.105045 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260926 20:10:26.106629 24429 tablet_service.cc:2908] Rejecting scan request for tablet c68f72b2b15c4fa8ad2e4ea41d5ec926: Uninitialized: clean time has not yet been initialized
I20260926 20:10:26.108570 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 24386
I20260926 20:10:26.114612 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 24522
I20260926 20:10:26.119992 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 24657
I20260926 20:10:26.125478 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 24292
2026-09-26T20:10:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2661 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:10:26 dist-test-slave-7034 krb5kdc[24828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:10:26 dist-test-slave-7034 krb5kdc[24828](info): set up 2 sockets
Sep 26 20:10:26 dist-test-slave-7034 krb5kdc[24828](info): commencing operation
krb5kdc: starting...
W20260926 20:10:27.166224 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:10:27 dist-test-slave-7034 krb5kdc[24828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:10:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:10:27.293659 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:46181
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:46181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:27.404947 24841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:27.405225 24841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:27.405273 24841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:27.409435 24841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:27.409494 24841 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:10:27.409511 24841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:27.409529 24841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:27.409590 24841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:27.409619 24841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:27.414949 24841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:46181
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:46181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:27.416002 24841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:27.417038 24841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:27.425643 24846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:27.425657 24841 server_base.cc:1034] running on GCE node
W20260926 20:10:27.425648 24847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:27.425706 24849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:27.426469 24841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:27.427552 24841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:27.428737 24841 hybrid_clock.cc:648] HybridClock initialized: now 1790453427428724 us; error 25 us; skew 500 ppm
Sep 26 20:10:27 dist-test-slave-7034 krb5kdc[24828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453427, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:27.435711 24841 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:10:27.436693 24841 webserver.cc:460] Webserver started at http://127.23.50.190:45431/ using document root <none> and password file <none>
I20260926 20:10:27.437280 24841 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:27.437371 24841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:27.437603 24841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:27.439724 24841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "e0489d5334224cbda92584143b5cf3ea"
format_stamp: "Formatted at 2026-09-26 20:10:27 on dist-test-slave-7034"
server_key: "276edc1bf1c5d2ddac05c11a599300fb"
server_key_iv: "363f03018dd15ea159c01ad1928816ed"
server_key_version: "encryptionkey@0"
I20260926 20:10:27.440305 24841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "e0489d5334224cbda92584143b5cf3ea"
format_stamp: "Formatted at 2026-09-26 20:10:27 on dist-test-slave-7034"
server_key: "276edc1bf1c5d2ddac05c11a599300fb"
server_key_iv: "363f03018dd15ea159c01ad1928816ed"
server_key_version: "encryptionkey@0"
I20260926 20:10:27.443990 24841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 20:10:27.446491 24856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:27.448010 24841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:27.448180 24841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "e0489d5334224cbda92584143b5cf3ea"
format_stamp: "Formatted at 2026-09-26 20:10:27 on dist-test-slave-7034"
server_key: "276edc1bf1c5d2ddac05c11a599300fb"
server_key_iv: "363f03018dd15ea159c01ad1928816ed"
server_key_version: "encryptionkey@0"
I20260926 20:10:27.448304 24841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:27.466276 24841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:27.469511 24841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:27.469728 24841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:27.477780 24841 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:46181
I20260926 20:10:27.477895 24919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:46181 every 8 connection(s)
I20260926 20:10:27.478811 24841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:27.480122 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 24841
I20260926 20:10:27.480235 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:27.480495 23754 external_mini_cluster.cc:1428] Setting key 0d44f631dbeff8f7862feb3073b92ad1
I20260926 20:10:27.482308 24920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:27.488978 24920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea: Bootstrap starting.
Sep 26 20:10:27 dist-test-slave-7034 krb5kdc[24828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:27.491349 24920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:27.492439 24920 log.cc:824] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:27.494630 24920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea: No bootstrap required, opened a new log
I20260926 20:10:27.494956 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:27.481822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.23.50.190:46181)
0926 20:10:27.482286 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:27.482302 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:27.482338 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:10:27.483003 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:27.483018 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:27.483348 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:27.483619 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:27.484469 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.485437 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.486586 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.487060 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.487178 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:27.490108 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:27.490113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:27.490126 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:27.490154 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:27.491952 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:27.492440 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:27.492446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:27.492451 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:27.492618 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:27.492911 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:27.492916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:27.492918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:27.493333 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:27.493536 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:27.493875 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 20:10:27.494173 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":156,"threads_started":1}
I20260926 20:10:27.498126 24920 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0489d5334224cbda92584143b5cf3ea" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46181 } }
I20260926 20:10:27.498536 24920 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:27.498610 24920 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0489d5334224cbda92584143b5cf3ea, State: Initialized, Role: FOLLOWER
I20260926 20:10:27.499181 24920 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0489d5334224cbda92584143b5cf3ea" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46181 } }
I20260926 20:10:27.499413 24920 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:27.499485 24920 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:27.499586 24920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:27.500824 24920 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0489d5334224cbda92584143b5cf3ea" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46181 } }
I20260926 20:10:27.501192 24920 leader_election.cc:302] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0489d5334224cbda92584143b5cf3ea; no voters:
I20260926 20:10:27.501569 24920 leader_election.cc:288] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:27.501713 24925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:27.501995 24925 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [term 1 LEADER]: Becoming Leader. State: Replica: e0489d5334224cbda92584143b5cf3ea, State: Running, Role: LEADER
I20260926 20:10:27.502350 24925 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0489d5334224cbda92584143b5cf3ea" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46181 } }
I20260926 20:10:27.502948 24920 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:27.504081 24927 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0489d5334224cbda92584143b5cf3ea. Latest consensus state: current_term: 1 leader_uuid: "e0489d5334224cbda92584143b5cf3ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0489d5334224cbda92584143b5cf3ea" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46181 } } }
I20260926 20:10:27.504209 24927 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:27.504494 24926 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0489d5334224cbda92584143b5cf3ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0489d5334224cbda92584143b5cf3ea" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46181 } } }
I20260926 20:10:27.504561 24934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:27.504593 24926 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:27.507922 24934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:27.514760 24934 catalog_manager.cc:1347] Generated new cluster ID: e6fdf63ebe4e4ec6a4c68399ff371a30
I20260926 20:10:27.514837 24934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:27.529428 24934 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:27.530483 24934 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:27.536301 24934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea: Generated new TSK 0
I20260926 20:10:27.537128 24934 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:10:27.573287 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:46181
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:27.688745 24946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:27.689097 24946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:27.689184 24946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:27.693352 24946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:27.693454 24946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:27.693529 24946 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:10:27.693615 24946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:10:27.699102 24946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.50.190:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:27.700291 24946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:27.701355 24946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:10:27.711974 24946 server_base.cc:1034] running on GCE node
W20260926 20:10:27.711923 24951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:27.711916 24952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:27.711923 24954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:27.712685 24946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:27.713404 24946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:27.714574 24946 hybrid_clock.cc:648] HybridClock initialized: now 1790453427714548 us; error 44 us; skew 500 ppm
Sep 26 20:10:27 dist-test-slave-7034 krb5kdc[24828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453427, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:27.717415 24946 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:10:27.718451 24946 webserver.cc:460] Webserver started at http://127.23.50.129:39297/ using document root <none> and password file <none>
I20260926 20:10:27.718982 24946 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:27.719039 24946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:27.719208 24946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:27.721244 24946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91"
format_stamp: "Formatted at 2026-09-26 20:10:27 on dist-test-slave-7034"
server_key: "7cc9a13d372c48e39b09a433887800b9"
server_key_iv: "96a6b7fa570c5d855ae49d511f7ffb56"
server_key_version: "encryptionkey@0"
I20260926 20:10:27.721830 24946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91"
format_stamp: "Formatted at 2026-09-26 20:10:27 on dist-test-slave-7034"
server_key: "7cc9a13d372c48e39b09a433887800b9"
server_key_iv: "96a6b7fa570c5d855ae49d511f7ffb56"
server_key_version: "encryptionkey@0"
I20260926 20:10:27.725711 24946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:10:27.728341 24961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:27.729568 24946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:27.729704 24946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91"
format_stamp: "Formatted at 2026-09-26 20:10:27 on dist-test-slave-7034"
server_key: "7cc9a13d372c48e39b09a433887800b9"
server_key_iv: "96a6b7fa570c5d855ae49d511f7ffb56"
server_key_version: "encryptionkey@0"
I20260926 20:10:27.729826 24946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:27.754735 24946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:27.758140 24946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:27.758381 24946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:27.759913 24946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:27.759990 24946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:27.760059 24946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:27.760123 24946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:10:27 dist-test-slave-7034 krb5kdc[24828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453427, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:27.771608 24946 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:36189
I20260926 20:10:27.771739 25077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:36189 every 8 connection(s)
I20260926 20:10:27.772711 24946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:10:27.774552 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:27.761030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:36545 (local address 127.23.50.190:46181)
0926 20:10:27.761161 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:27.761165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:27.762320 (+ 1155us) server_negotiation.cc:407] Connection header received
0926 20:10:27.763592 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:27.763596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:27.763667 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:27.763768 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:27.765095 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.765698 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.766924 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.767261 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.767386 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:27.770484 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:27.770487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:27.770489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:27.770515 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:27.772145 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:27.772743 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:27.772746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:27.772747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:27.772861 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:27.773283 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:27.773287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:27.773289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:27.773547 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:27.773679 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:27.774251 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 20:10:27.774383 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:10:27.775405 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:27.761611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:46181 (local address 127.23.50.129:36545)
0926 20:10:27.762158 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:27.762202 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:27.763333 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:27.763952 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:27.763966 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:27.764318 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:27.764948 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:27.764963 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.765825 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.765829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:27.766793 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:27.766803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.767436 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.767440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:27.767612 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:27.768290 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:27.768318 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:27.770247 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:27.772257 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:27.772263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:27.772278 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:27.772600 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:27.772963 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:27.772966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:27.772968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:27.773160 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:27.773681 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:27.773689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:27.774007 (+ 318us) client_negotiation.cc:769] Sending connection context
0926 20:10:27.774201 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 20:10:27.774519 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":168,"threads_started":1}
I20260926 20:10:27.776791 25078 heartbeater.cc:347] Connected to a master server at 127.23.50.190:46181
I20260926 20:10:27.776949 25078 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:27.777529 25078 heartbeater.cc:511] Master 127.23.50.190:46181 requested a full tablet report, sending...
I20260926 20:10:27.779242 24873 ts_manager.cc:194] Registered new tserver with Master: f94d5ca0d1cc40fda2fcd5ebd417ad91 (127.23.50.129:36189)
I20260926 20:10:27.780527 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 24946
I20260926 20:10:27.780670 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:10:27.780762 24873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:36545
I20260926 20:10:27.781003 23754 external_mini_cluster.cc:1428] Setting key 56e38b171d0662c9b1238e19a2522a93
I20260926 20:10:27.787184 25084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:27.779016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:46181 (local address 127.23.50.129:51855)
0926 20:10:27.779326 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:27.779339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:27.779452 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:27.779695 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:27.779697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:27.779862 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:27.779956 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:27.779961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.780970 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.780973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:27.781784 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:27.781793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.782460 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.782464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:27.782554 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:27.783113 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:27.783133 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:27.783623 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:27.785418 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:27.785420 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:27.785422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:27.785611 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:27.785935 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:27.785938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:27.785939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:27.786052 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:27.786540 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:27.786543 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:27.786629 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 20:10:27.786815 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 20:10:27.786964 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260926 20:10:27.787192 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:27.779099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:51855 (local address 127.23.50.190:46181)
0926 20:10:27.779228 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:27.779232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:27.779412 (+ 180us) server_negotiation.cc:407] Connection header received
0926 20:10:27.779540 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:27.779543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:27.779583 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:27.779668 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:27.780061 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.780817 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.781941 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.782227 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.782319 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:27.783770 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:27.783773 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:27.783775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:27.783796 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:27.785318 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:27.785712 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:27.785719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:27.785723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:27.785815 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:27.786151 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:27.786156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:27.786161 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:27.786409 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:27.786518 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:27.786880 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 20:10:27.787015 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:10:27.810413 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:46181
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:27.924579 25087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:27.924870 25087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:27.924916 25087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:27.929102 25087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:27.929183 25087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:27.929224 25087 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:10:27.929260 25087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:10:27.935171 25087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.50.190:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:27.936265 25087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:27.937237 25087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:27.947369 25093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:27.947418 25095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:27.947418 25092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:27.947566 25087 server_base.cc:1034] running on GCE node
I20260926 20:10:27.948211 25087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:27.948935 25087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:27.950133 25087 hybrid_clock.cc:648] HybridClock initialized: now 1790453427950123 us; error 28 us; skew 500 ppm
Sep 26 20:10:27 dist-test-slave-7034 krb5kdc[24828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453427, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:27.953254 25087 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:10:27.954252 25087 webserver.cc:460] Webserver started at http://127.23.50.130:34329/ using document root <none> and password file <none>
I20260926 20:10:27.954767 25087 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:27.954824 25087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:27.954984 25087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:27.957034 25087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "ffe3cdf8471b4e1c958dbda73b75ed01"
format_stamp: "Formatted at 2026-09-26 20:10:27 on dist-test-slave-7034"
server_key: "7194f222699111add0c01ca62a62abe6"
server_key_iv: "0431c3e98893720ebbad6d750a5da1ff"
server_key_version: "encryptionkey@0"
I20260926 20:10:27.957543 25087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "ffe3cdf8471b4e1c958dbda73b75ed01"
format_stamp: "Formatted at 2026-09-26 20:10:27 on dist-test-slave-7034"
server_key: "7194f222699111add0c01ca62a62abe6"
server_key_iv: "0431c3e98893720ebbad6d750a5da1ff"
server_key_version: "encryptionkey@0"
I20260926 20:10:27.961241 25087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 20:10:27.963953 25102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:27.965422 25087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:10:27.965538 25087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "ffe3cdf8471b4e1c958dbda73b75ed01"
format_stamp: "Formatted at 2026-09-26 20:10:27 on dist-test-slave-7034"
server_key: "7194f222699111add0c01ca62a62abe6"
server_key_iv: "0431c3e98893720ebbad6d750a5da1ff"
server_key_version: "encryptionkey@0"
I20260926 20:10:27.965644 25087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:27.978559 25087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:27.981977 25087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:27.982136 25087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:27.983606 25087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:27.983656 25087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:27.983700 25087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:27.983747 25087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:10:27 dist-test-slave-7034 krb5kdc[24828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453427, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:27.994342 25087 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:35279
I20260926 20:10:27.995419 25087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:10:27.996136 25218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:35279 every 8 connection(s)
I20260926 20:10:27.996847 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 25087
I20260926 20:10:27.996966 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:10:27.997323 23754 external_mini_cluster.cc:1428] Setting key 5bbed80843bb3b87faea368c004881cc
I20260926 20:10:27.998085 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:27.984421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:46673 (local address 127.23.50.190:46181)
0926 20:10:27.984571 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:27.984576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:27.985460 (+ 884us) server_negotiation.cc:407] Connection header received
0926 20:10:27.986717 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:27.986722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:27.986789 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:27.986896 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:27.988278 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.989175 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.990543 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.990824 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.990945 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:27.994023 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:27.994028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:27.994030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:27.994079 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:27.995701 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:27.996341 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:27.996347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:27.996349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:27.996455 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:27.996816 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:27.996820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:27.996822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:27.997066 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:27.997167 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:27.997760 (+ 593us) server_negotiation.cc:299] Negotiation successful
0926 20:10:27.997931 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:10:27.998971 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:27.984846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:46181 (local address 127.23.50.130:46673)
0926 20:10:27.985294 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:27.985345 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:27.986368 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:27.987112 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:27.987123 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:27.987576 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:27.988139 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:27.988154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.989351 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.989355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:27.990405 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:27.990416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:27.990993 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:27.990997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:27.991162 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:27.992319 (+ 1157us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:27.992342 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:27.993806 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:27.995838 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:27.995844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:27.995860 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:27.996184 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:27.996554 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:27.996563 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:27.996564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:27.996708 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:27.997180 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:27.997185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:27.997494 (+ 309us) client_negotiation.cc:769] Sending connection context
0926 20:10:27.997646 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 20:10:27.998016 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":133,"threads_started":1}
I20260926 20:10:28.000214 25219 heartbeater.cc:347] Connected to a master server at 127.23.50.190:46181
I20260926 20:10:28.000389 25219 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:28.000998 25219 heartbeater.cc:511] Master 127.23.50.190:46181 requested a full tablet report, sending...
I20260926 20:10:28.002182 24873 ts_manager.cc:194] Registered new tserver with Master: ffe3cdf8471b4e1c958dbda73b75ed01 (127.23.50.130:35279)
I20260926 20:10:28.002854 24873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:46673
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:10:28.011125 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.002451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:38287 (local address 127.23.50.190:46181)
0926 20:10:28.002553 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.002557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.003414 (+ 857us) server_negotiation.cc:407] Connection header received
0926 20:10:28.003456 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.003459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.003529 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.003623 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:28.004172 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.004902 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.005841 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.006288 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.006416 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.007713 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:28.007717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:28.007720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:28.007746 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:28.009466 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:28.009964 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:28.009968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:28.009969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:28.010050 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:28.010325 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:28.010327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:28.010328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:28.010530 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:28.010607 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.010888 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.010982 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:10:28.011233 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.002411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:46181 (local address 127.23.50.130:38287)
0926 20:10:28.003229 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:28.003241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:28.003367 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:28.003651 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:28.003656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:28.003893 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:28.004039 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.004051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.005103 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.005106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.005713 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.005722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.006438 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.006441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.006528 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.007138 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:28.007151 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:28.007560 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:28.009579 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:28.009581 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:28.009583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:28.009794 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:28.010136 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:28.010138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:28.010140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:28.010239 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:28.010644 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:28.010647 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:28.010737 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 20:10:28.010833 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 20:10:28.010954 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:10:28.029098 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:46181
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:28.142659 25228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:28.142913 25228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:28.142958 25228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:28.147037 25228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:28.147091 25228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:28.147131 25228 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:10:28.147163 25228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:10:28.152328 25228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.50.190:46181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:28.153354 25228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:28.154389 25228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:28.164775 25234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:28.164781 25233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:28.164836 25228 server_base.cc:1034] running on GCE node
W20260926 20:10:28.164777 25236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:28.165410 25228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:28.166139 25228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:28.167323 25228 hybrid_clock.cc:648] HybridClock initialized: now 1790453428167286 us; error 54 us; skew 500 ppm
Sep 26 20:10:28 dist-test-slave-7034 krb5kdc[24828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453428, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:28.170251 25228 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:10:28.171207 25228 webserver.cc:460] Webserver started at http://127.23.50.131:42837/ using document root <none> and password file <none>
I20260926 20:10:28.171805 25228 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:28.171869 25228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:28.172092 25228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:28.174199 25228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "2c0c9fb29b5c430b995e1f6e310951cd"
format_stamp: "Formatted at 2026-09-26 20:10:28 on dist-test-slave-7034"
server_key: "d54b94181fc6e000f72dc95b894462a1"
server_key_iv: "1a4db8c4ce3889c28f695a5bf672d15b"
server_key_version: "encryptionkey@0"
I20260926 20:10:28.174767 25228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "2c0c9fb29b5c430b995e1f6e310951cd"
format_stamp: "Formatted at 2026-09-26 20:10:28 on dist-test-slave-7034"
server_key: "d54b94181fc6e000f72dc95b894462a1"
server_key_iv: "1a4db8c4ce3889c28f695a5bf672d15b"
server_key_version: "encryptionkey@0"
I20260926 20:10:28.178637 25228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:10:28.181020 25244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:28.182355 25228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:10:28.182482 25228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "2c0c9fb29b5c430b995e1f6e310951cd"
format_stamp: "Formatted at 2026-09-26 20:10:28 on dist-test-slave-7034"
server_key: "d54b94181fc6e000f72dc95b894462a1"
server_key_iv: "1a4db8c4ce3889c28f695a5bf672d15b"
server_key_version: "encryptionkey@0"
I20260926 20:10:28.182585 25228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:28.199708 25228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:28.202834 25228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:28.203027 25228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:28.204586 25228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:28.204635 25228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:28.204701 25228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:28.204741 25228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:10:28 dist-test-slave-7034 krb5kdc[24828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453428, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:28.215759 25228 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:35353
I20260926 20:10:28.215801 25360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:35353 every 8 connection(s)
I20260926 20:10:28.216777 25228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:10:28.218925 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.205575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:38133 (local address 127.23.50.190:46181)
0926 20:10:28.205702 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.205706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.206840 (+ 1134us) server_negotiation.cc:407] Connection header received
0926 20:10:28.208067 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.208071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.208153 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.208274 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:28.209702 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.210637 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.211981 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.212221 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.212340 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.215060 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:28.215064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:28.215067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:28.215103 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:28.216682 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:28.217307 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:28.217310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:28.217313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:28.217396 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:28.217764 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:28.217770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:28.217775 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:28.218061 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:28.218157 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.218643 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.218767 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:10:28.219707 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.206161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:46181 (local address 127.23.50.131:38133)
0926 20:10:28.206666 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:28.206707 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:28.207810 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:28.208521 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:28.208530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:28.208918 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:28.209558 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.209573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.210800 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.210804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.211853 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.211863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.212392 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.212396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.212566 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.213240 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:28.213262 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:28.214881 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:28.216811 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:28.216817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:28.216832 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:28.217185 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:28.217497 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:28.217500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:28.217502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:28.217665 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:28.218154 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:28.218160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:28.218408 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 20:10:28.218600 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 20:10:28.218926 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":153,"threads_started":1}
I20260926 20:10:28.220948 25361 heartbeater.cc:347] Connected to a master server at 127.23.50.190:46181
I20260926 20:10:28.221089 25361 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:28.221668 25361 heartbeater.cc:511] Master 127.23.50.190:46181 requested a full tablet report, sending...
I20260926 20:10:28.222954 24873 ts_manager.cc:194] Registered new tserver with Master: 2c0c9fb29b5c430b995e1f6e310951cd (127.23.50.131:35353)
I20260926 20:10:28.223548 24873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:38133
I20260926 20:10:28.225970 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 25228
I20260926 20:10:28.226080 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:10:28.226336 23754 external_mini_cluster.cc:1428] Setting key ff61be3235ecca2add07e371a36e488b
I20260926 20:10:28.228629 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:10:28.231931 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.224120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:36799 (local address 127.23.50.190:46181)
0926 20:10:28.224252 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.224258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.224525 (+ 267us) server_negotiation.cc:407] Connection header received
0926 20:10:28.224658 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.224661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.224699 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.224757 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:28.225243 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.225733 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.226707 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.226974 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.227077 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.228436 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:28.228440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:28.228442 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:28.228465 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:28.230153 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:28.230640 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:28.230647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:28.230650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:28.230764 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:28.231093 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:28.231096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:28.231097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:28.231265 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:28.231351 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.231707 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.231816 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 20:10:28.232086 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.224053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:46181 (local address 127.23.50.131:36799)
0926 20:10:28.224437 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:28.224452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:28.224548 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:28.224811 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:28.224814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:28.225023 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:28.225135 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.225143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.225886 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.225889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.226593 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.226602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.227090 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.227094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.227217 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.227801 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:28.227815 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:28.228310 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:28.230273 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:28.230276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:28.230277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:28.230522 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:28.230877 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:28.230882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:28.230885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:28.230996 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:28.231368 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:28.231372 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:28.231529 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 20:10:28.231646 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:10:28.231835 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":178,"threads_started":1}
Sep 26 20:10:28 dist-test-slave-7034 krb5kdc[24828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453428, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:10:28 dist-test-slave-7034 krb5kdc[24828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453428, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:28.256791 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.248145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.23.50.190:46181)
0926 20:10:28.248299 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.248303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.248316 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:10:28.248363 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.248365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.248406 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.248497 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:28.249273 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.249818 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.250796 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.251017 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.251105 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.253896 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:28.253906 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:28.253909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:28.253936 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:28.255223 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:28.255655 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:28.255658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:28.255660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:28.255739 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:28.256050 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:28.256052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:28.256053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:28.256228 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:28.256387 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.256550 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.256651 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:10:28.259963 24873 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:10:28.263134 24873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:28.282982 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.276042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36189 (local address 127.0.0.1:53654)
0926 20:10:28.276935 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:28.276949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:28.277143 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:28.277862 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:28.277868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:28.277891 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:28.278091 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.278099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.279548 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.279552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.281008 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.281018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.282383 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.282387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.282496 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.282535 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 20:10:28.282653 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 20:10:28.282797 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":25,"thread_start_us":92,"threads_started":1}
I20260926 20:10:28.283627 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.276248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.23.50.129:36189)
0926 20:10:28.276658 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.276667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.277068 (+ 401us) server_negotiation.cc:407] Connection header received
0926 20:10:28.277289 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.277296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.277524 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.277666 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:28.278223 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.279419 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.281142 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.282062 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.282166 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.282317 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.283373 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.283485 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":84,"threads_started":1}
I20260926 20:10:28.284231 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.276486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:35279 (local address 127.0.0.1:51320)
0926 20:10:28.277353 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:28.277364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:28.277502 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:28.278181 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:28.278186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:28.278204 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:28.278339 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.278345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.280541 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.280544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.281687 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.281697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.283021 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.283026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.283881 (+ 855us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.283905 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:10:28.284010 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 20:10:28.284086 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":47,"thread_start_us":86,"threads_started":1}
I20260926 20:10:28.284287 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.276800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.23.50.130:35279)
0926 20:10:28.277187 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.277198 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.277438 (+ 240us) server_negotiation.cc:407] Connection header received
0926 20:10:28.277647 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.277652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.277890 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.278017 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:28.278477 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.280400 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.281831 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.282694 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.282770 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.282919 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.284046 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.284139 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":90,"threads_started":1}
I20260926 20:10:28.285786 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.275309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:35353 (local address 127.0.0.1:38584)
0926 20:10:28.275955 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:28.276016 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:28.276154 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:28.277925 (+ 1771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:28.277930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:28.277957 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:28.278434 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.278445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.280054 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.280060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.281832 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.281878 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.283474 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.283480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.283626 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.283664 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:10:28.285469 (+ 1805us) client_negotiation.cc:240] Negotiation successful
0926 20:10:28.285583 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":250,"thread_start_us":92,"threads_started":1}
I20260926 20:10:28.286476 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.276779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.23.50.131:35353)
0926 20:10:28.277367 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.277380 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.277402 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:10:28.277476 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.277484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.277695 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.278040 (+ 345us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:28.278603 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.279889 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.282034 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.283100 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.283225 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.283397 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.286116 (+ 2719us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.286269 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":111,"threads_started":1}
I20260926 20:10:28.286938 25012 tablet_service.cc:1467] Processing CreateTablet for tablet e1f9420a71704e4697f09c6d3cc9638e (DEFAULT_TABLE table=test-table [id=714c69c461414529853713db1df6ac1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:28.287489 25152 tablet_service.cc:1467] Processing CreateTablet for tablet e1f9420a71704e4697f09c6d3cc9638e (DEFAULT_TABLE table=test-table [id=714c69c461414529853713db1df6ac1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:28.288398 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f9420a71704e4697f09c6d3cc9638e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:28.288553 25152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f9420a71704e4697f09c6d3cc9638e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:28.289706 25290 tablet_service.cc:1467] Processing CreateTablet for tablet e1f9420a71704e4697f09c6d3cc9638e (DEFAULT_TABLE table=test-table [id=714c69c461414529853713db1df6ac1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:28.290920 25290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f9420a71704e4697f09c6d3cc9638e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:28.298206 25384 tablet_bootstrap.cc:492] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91: Bootstrap starting.
I20260926 20:10:28.300348 25384 tablet_bootstrap.cc:654] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:28.300472 25386 tablet_bootstrap.cc:492] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd: Bootstrap starting.
I20260926 20:10:28.301241 25384 log.cc:824] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:28.301496 25385 tablet_bootstrap.cc:492] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01: Bootstrap starting.
I20260926 20:10:28.302567 25386 tablet_bootstrap.cc:654] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:28.304018 25386 log.cc:824] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:28.304147 25385 tablet_bootstrap.cc:654] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:28.305135 25385 log.cc:824] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:28.306036 25386 tablet_bootstrap.cc:492] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd: No bootstrap required, opened a new log
I20260926 20:10:28.306258 25386 ts_tablet_manager.cc:1397] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:10:28.307380 25385 tablet_bootstrap.cc:492] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01: No bootstrap required, opened a new log
I20260926 20:10:28.307582 25385 ts_tablet_manager.cc:1397] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:10:28.309406 25386 raft_consensus.cc:348] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } }
I20260926 20:10:28.309650 25386 raft_consensus.cc:374] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:28.309711 25386 raft_consensus.cc:729] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0c9fb29b5c430b995e1f6e310951cd, State: Initialized, Role: FOLLOWER
I20260926 20:10:28.310182 25386 consensus_queue.cc:260] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } }
I20260926 20:10:28.310854 25361 heartbeater.cc:503] Master 127.23.50.190:46181 was elected leader, sending a full tablet report...
I20260926 20:10:28.312958 25385 raft_consensus.cc:348] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } }
I20260926 20:10:28.313292 25385 raft_consensus.cc:374] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:28.313354 25385 raft_consensus.cc:729] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe3cdf8471b4e1c958dbda73b75ed01, State: Initialized, Role: FOLLOWER
I20260926 20:10:28.313948 25386 ts_tablet_manager.cc:1428] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260926 20:10:28.315738 25384 tablet_bootstrap.cc:492] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91: No bootstrap required, opened a new log
I20260926 20:10:28.315989 25384 ts_tablet_manager.cc:1397] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20260926 20:10:28.317991 25385 consensus_queue.cc:260] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } }
I20260926 20:10:28.318697 25219 heartbeater.cc:503] Master 127.23.50.190:46181 was elected leader, sending a full tablet report...
I20260926 20:10:28.319087 25385 ts_tablet_manager.cc:1428] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260926 20:10:28.322006 25384 raft_consensus.cc:348] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } }
I20260926 20:10:28.322449 25384 raft_consensus.cc:374] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:28.322521 25384 raft_consensus.cc:729] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94d5ca0d1cc40fda2fcd5ebd417ad91, State: Initialized, Role: FOLLOWER
I20260926 20:10:28.323077 25384 consensus_queue.cc:260] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } }
I20260926 20:10:28.323768 25078 heartbeater.cc:503] Master 127.23.50.190:46181 was elected leader, sending a full tablet report...
I20260926 20:10:28.324276 25384 ts_tablet_manager.cc:1428] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260926 20:10:28.357455 25390 raft_consensus.cc:482] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:28.357678 25390 raft_consensus.cc:504] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } }
I20260926 20:10:28.358872 25390 leader_election.cc:288] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe3cdf8471b4e1c958dbda73b75ed01 (127.23.50.130:35279), f94d5ca0d1cc40fda2fcd5ebd417ad91 (127.23.50.129:36189)
I20260926 20:10:28.364483 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.359120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36189 (local address 127.23.50.131:56093)
0926 20:10:28.359334 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:28.359349 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:28.359440 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:28.359828 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:28.359830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:28.359843 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:28.359932 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.359936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.360923 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.360927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.362980 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.362991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.364062 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.364065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.364163 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.364183 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 20:10:28.364275 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:10:28.364348 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":203}
I20260926 20:10:28.364482 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.359249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:56093 (local address 127.23.50.129:36189)
0926 20:10:28.359377 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.359384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.359401 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:10:28.359575 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.359578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.359638 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.359706 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:28.360049 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.360798 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.363118 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.363938 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.364092 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.364191 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.364300 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.364376 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:10:28.365407 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f9420a71704e4697f09c6d3cc9638e" candidate_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" is_pre_election: true
I20260926 20:10:28.365638 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.359146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:35279 (local address 127.23.50.131:46425)
0926 20:10:28.359589 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:28.359602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:28.359678 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:28.360078 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:28.360080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:28.360097 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:28.360206 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.360211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.361427 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.361431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.362980 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.362991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.365191 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.365195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.365288 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.365309 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:10:28.365384 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:10:28.365456 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":83,"thread_start_us":128,"threads_started":1}
I20260926 20:10:28.365695 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.359618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:46425 (local address 127.23.50.130:35279)
0926 20:10:28.359761 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.359768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.359785 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:10:28.359848 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.359853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.359927 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.360012 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:28.360333 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.361299 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.364260 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.365034 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.365184 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.365256 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.365422 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.365521 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:10:28.365757 25032 raft_consensus.cc:2454] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c0c9fb29b5c430b995e1f6e310951cd in term 0.
I20260926 20:10:28.366315 25245 leader_election.cc:302] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c0c9fb29b5c430b995e1f6e310951cd, f94d5ca0d1cc40fda2fcd5ebd417ad91; no voters:
I20260926 20:10:28.366457 25173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f9420a71704e4697f09c6d3cc9638e" candidate_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" is_pre_election: true
I20260926 20:10:28.366601 25390 raft_consensus.cc:2781] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:28.366675 25390 raft_consensus.cc:482] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:28.366705 25390 raft_consensus.cc:3037] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:28.366767 25173 raft_consensus.cc:2454] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c0c9fb29b5c430b995e1f6e310951cd in term 0.
I20260926 20:10:28.367851 25390 raft_consensus.cc:504] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } }
I20260926 20:10:28.368240 25390 leader_election.cc:288] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [CANDIDATE]: Term 1 election: Requested vote from peers ffe3cdf8471b4e1c958dbda73b75ed01 (127.23.50.130:35279), f94d5ca0d1cc40fda2fcd5ebd417ad91 (127.23.50.129:36189)
I20260926 20:10:28.368575 25173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f9420a71704e4697f09c6d3cc9638e" candidate_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01"
I20260926 20:10:28.368633 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f9420a71704e4697f09c6d3cc9638e" candidate_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91"
I20260926 20:10:28.368721 25173 raft_consensus.cc:3037] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:28.368767 25032 raft_consensus.cc:3037] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:28.370092 25032 raft_consensus.cc:2454] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c0c9fb29b5c430b995e1f6e310951cd in term 1.
I20260926 20:10:28.370980 25173 raft_consensus.cc:2454] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c0c9fb29b5c430b995e1f6e310951cd in term 1.
I20260926 20:10:28.371328 25245 leader_election.cc:302] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c0c9fb29b5c430b995e1f6e310951cd, f94d5ca0d1cc40fda2fcd5ebd417ad91; no voters:
I20260926 20:10:28.371579 25390 raft_consensus.cc:2781] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:28.371862 25390 raft_consensus.cc:686] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 1 LEADER]: Becoming Leader. State: Replica: 2c0c9fb29b5c430b995e1f6e310951cd, State: Running, Role: LEADER
I20260926 20:10:28.372217 25390 consensus_queue.cc:237] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } }
I20260926 20:10:28.376292 24873 catalog_manager.cc:5495] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c0c9fb29b5c430b995e1f6e310951cd (127.23.50.131). New cstate: current_term: 1 leader_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } health_report { overall_health: UNKNOWN } } }
I20260926 20:10:28.412355 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.408092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.23.50.131:35353)
0926 20:10:28.408235 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.408240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.408255 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:10:28.408468 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.408471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.408526 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.408631 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:28.408918 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.409430 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.410443 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.410681 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.410793 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.411219 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:28.411822 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:28.411958 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.412054 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.412139 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:10:28.419205 25173 raft_consensus.cc:1261] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01 [term 1 FOLLOWER]: Refusing update from remote peer 2c0c9fb29b5c430b995e1f6e310951cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:28.419205 25032 raft_consensus.cc:1261] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 1 FOLLOWER]: Refusing update from remote peer 2c0c9fb29b5c430b995e1f6e310951cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:28.419957 25394 consensus_queue.cc:1035] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:28.420177 25390 consensus_queue.cc:1035] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:28.428422 25397 mvcc.cc:205] Tried to move back new op lower bound from 7333697242797600768 to 7333697242616037376. Current Snapshot: MvccSnapshot[applied={T|T < 7333697242797600768}]
I20260926 20:10:28.430548 25398 mvcc.cc:205] Tried to move back new op lower bound from 7333697242797600768 to 7333697242616037376. Current Snapshot: MvccSnapshot[applied={T|T < 7333697242797600768}]
I20260926 20:10:28.434592 25399 mvcc.cc:205] Tried to move back new op lower bound from 7333697242797600768 to 7333697242616037376. Current Snapshot: MvccSnapshot[applied={T|T < 7333697242797600768 or (T in {7333697242797600768})}]
I20260926 20:10:28.444263 24874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:28.444454 24874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:28.447551 24874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea: Sending DeleteTablet for 3 replicas of tablet e1f9420a71704e4697f09c6d3cc9638e
I20260926 20:10:28.448583 25152 tablet_service.cc:1514] Processing DeleteTablet for tablet e1f9420a71704e4697f09c6d3cc9638e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:28 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:51320
I20260926 20:10:28.448599 25290 tablet_service.cc:1514] Processing DeleteTablet for tablet e1f9420a71704e4697f09c6d3cc9638e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:28 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:38584
I20260926 20:10:28.449168 25406 tablet_replica.cc:330] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01: stopping tablet replica
I20260926 20:10:28.449528 25406 raft_consensus.cc:2229] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:28.449926 25406 raft_consensus.cc:2258] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:28.449900 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet e1f9420a71704e4697f09c6d3cc9638e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:28 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:53654
I20260926 20:10:28.451509 24874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:10:28.451710 25406 ts_tablet_manager.cc:1905] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:10:28.452061 24874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:28.454569 25406 ts_tablet_manager.cc:1918] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:10:28.454666 25406 log.cc:1196] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/e1f9420a71704e4697f09c6d3cc9638e
I20260926 20:10:28.454963 25406 ts_tablet_manager.cc:1939] T e1f9420a71704e4697f09c6d3cc9638e P ffe3cdf8471b4e1c958dbda73b75ed01: Deleting consensus metadata
I20260926 20:10:28.456259 24859 catalog_manager.cc:4841] TS ffe3cdf8471b4e1c958dbda73b75ed01 (127.23.50.130:35279): tablet e1f9420a71704e4697f09c6d3cc9638e (table test-table [id=714c69c461414529853713db1df6ac1f]) successfully deleted
I20260926 20:10:28.458137 25407 tablet_replica.cc:330] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd: stopping tablet replica
I20260926 20:10:28.458534 25408 tablet_replica.cc:330] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91: stopping tablet replica
I20260926 20:10:28.458950 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 34518f01688946a5bf316b6d9a5d1e50 (DEFAULT_TABLE table=test-table [id=afce46b24adf4d76a0ac1df856c72194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:28.459230 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34518f01688946a5bf316b6d9a5d1e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:28.460382 25407 raft_consensus.cc:2229] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:10:28.458993 25408 raft_consensus.cc:2229] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:28.460929 25408 raft_consensus.cc:2258] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:28.461472 25407 raft_consensus.cc:2258] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:28.462150 25384 tablet_bootstrap.cc:492] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91: Bootstrap starting.
I20260926 20:10:28.461958 25290 tablet_service.cc:1467] Processing CreateTablet for tablet 34518f01688946a5bf316b6d9a5d1e50 (DEFAULT_TABLE table=test-table [id=afce46b24adf4d76a0ac1df856c72194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:28.463088 25290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34518f01688946a5bf316b6d9a5d1e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:28.463156 25384 tablet_bootstrap.cc:654] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:28.464370 25152 tablet_service.cc:1467] Processing CreateTablet for tablet 34518f01688946a5bf316b6d9a5d1e50 (DEFAULT_TABLE table=test-table [id=afce46b24adf4d76a0ac1df856c72194]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:28.464607 25152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34518f01688946a5bf316b6d9a5d1e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:28.466182 25386 tablet_bootstrap.cc:492] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd: Bootstrap starting.
I20260926 20:10:28.467123 25386 tablet_bootstrap.cc:654] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:28.467814 25385 tablet_bootstrap.cc:492] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01: Bootstrap starting.
I20260926 20:10:28.468420 25384 tablet_bootstrap.cc:492] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91: No bootstrap required, opened a new log
I20260926 20:10:28.468513 25384 ts_tablet_manager.cc:1397] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 20:10:28.468823 25385 tablet_bootstrap.cc:654] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:28.469046 25384 raft_consensus.cc:348] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.469193 25384 raft_consensus.cc:374] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:28.469235 25384 raft_consensus.cc:729] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94d5ca0d1cc40fda2fcd5ebd417ad91, State: Initialized, Role: FOLLOWER
I20260926 20:10:28.469390 25384 consensus_queue.cc:260] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.469658 25384 ts_tablet_manager.cc:1428] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 20:10:28.471660 25408 ts_tablet_manager.cc:1905] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:28.474609 25407 ts_tablet_manager.cc:1905] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:28.477077 25386 tablet_bootstrap.cc:492] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd: No bootstrap required, opened a new log
I20260926 20:10:28.477185 25386 ts_tablet_manager.cc:1397] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260926 20:10:28.477710 25386 raft_consensus.cc:348] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.477828 25386 raft_consensus.cc:374] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:28.477886 25386 raft_consensus.cc:729] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0c9fb29b5c430b995e1f6e310951cd, State: Initialized, Role: FOLLOWER
I20260926 20:10:28.478024 25386 consensus_queue.cc:260] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.478071 25407 ts_tablet_manager.cc:1918] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:10:28.478170 25407 log.cc:1196] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/e1f9420a71704e4697f09c6d3cc9638e
I20260926 20:10:28.478322 25386 ts_tablet_manager.cc:1428] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:10:28.478489 25407 ts_tablet_manager.cc:1939] T e1f9420a71704e4697f09c6d3cc9638e P 2c0c9fb29b5c430b995e1f6e310951cd: Deleting consensus metadata
I20260926 20:10:28.479221 25385 tablet_bootstrap.cc:492] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01: No bootstrap required, opened a new log
I20260926 20:10:28.479315 25385 ts_tablet_manager.cc:1397] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260926 20:10:28.479816 25408 ts_tablet_manager.cc:1918] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:10:28.479830 25385 raft_consensus.cc:348] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.479951 25385 raft_consensus.cc:374] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:28.479986 25385 raft_consensus.cc:729] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe3cdf8471b4e1c958dbda73b75ed01, State: Initialized, Role: FOLLOWER
I20260926 20:10:28.480089 25408 log.cc:1196] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/wal/wals/e1f9420a71704e4697f09c6d3cc9638e
I20260926 20:10:28.480122 25385 consensus_queue.cc:260] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.480432 25385 ts_tablet_manager.cc:1428] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 20:10:28.480582 25408 ts_tablet_manager.cc:1939] T e1f9420a71704e4697f09c6d3cc9638e P f94d5ca0d1cc40fda2fcd5ebd417ad91: Deleting consensus metadata
I20260926 20:10:28.480947 24857 catalog_manager.cc:4841] TS 2c0c9fb29b5c430b995e1f6e310951cd (127.23.50.131:35353): tablet e1f9420a71704e4697f09c6d3cc9638e (table test-table [id=714c69c461414529853713db1df6ac1f]) successfully deleted
I20260926 20:10:28.481971 24857 catalog_manager.cc:4841] TS f94d5ca0d1cc40fda2fcd5ebd417ad91 (127.23.50.129:36189): tablet e1f9420a71704e4697f09c6d3cc9638e (table test-table [id=714c69c461414529853713db1df6ac1f]) successfully deleted
W20260926 20:10:28.496611 25220 tablet.cc:2369] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:10:28.525440 25079 tablet.cc:2369] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:10:28.532281 25392 raft_consensus.cc:482] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:28.532419 25392 raft_consensus.cc:504] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.533689 25392 leader_election.cc:288] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe3cdf8471b4e1c958dbda73b75ed01 (127.23.50.130:35279), 2c0c9fb29b5c430b995e1f6e310951cd (127.23.50.131:35353)
I20260926 20:10:28.538513 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.534322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:57163 (local address 127.23.50.130:35279)
0926 20:10:28.534469 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.534473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.534491 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:10:28.534661 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.534664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.534712 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.534793 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:28.535127 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.535759 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.537238 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.537869 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.537986 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.538040 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.538289 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.538360 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:10:28.538610 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.534076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:42213 (local address 127.23.50.131:35353)
0926 20:10:28.534229 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.534236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.534908 (+ 672us) server_negotiation.cc:407] Connection header received
0926 20:10:28.535005 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.535008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.535051 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.535120 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:28.535460 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.536209 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.537536 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.538051 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.538165 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.538246 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.538423 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.538491 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:10:28.538623 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.534467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:35353 (local address 127.23.50.129:42213)
0926 20:10:28.534829 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:28.534841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:28.534926 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:28.535155 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:28.535158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:28.535175 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:28.535303 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.535309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.536328 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.536332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.537415 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.537423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.538188 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.538192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.538297 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.538319 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:10:28.538396 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:10:28.538462 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":40,"thread_start_us":89,"threads_started":1}
I20260926 20:10:28.539280 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.534015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:35279 (local address 127.23.50.129:57163)
0926 20:10:28.534414 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:28.534430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:28.534538 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:28.534829 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:28.534833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:28.534859 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:28.535006 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.535012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.535873 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.535876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.537095 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.537106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.538005 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.538009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.538133 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.538159 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:10:28.539061 (+ 902us) client_negotiation.cc:240] Negotiation successful
0926 20:10:28.539137 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":44,"thread_start_us":91,"threads_started":1}
I20260926 20:10:28.539395 25314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34518f01688946a5bf316b6d9a5d1e50" candidate_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" is_pre_election: true
I20260926 20:10:28.539662 25314 raft_consensus.cc:2454] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f94d5ca0d1cc40fda2fcd5ebd417ad91 in term 0.
I20260926 20:10:28.539950 25173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34518f01688946a5bf316b6d9a5d1e50" candidate_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" is_pre_election: true
I20260926 20:10:28.540113 25173 raft_consensus.cc:2454] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f94d5ca0d1cc40fda2fcd5ebd417ad91 in term 0.
I20260926 20:10:28.540371 24962 leader_election.cc:302] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c0c9fb29b5c430b995e1f6e310951cd, f94d5ca0d1cc40fda2fcd5ebd417ad91; no voters:
I20260926 20:10:28.540644 25392 raft_consensus.cc:2781] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:28.540711 25392 raft_consensus.cc:482] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:28.540750 25392 raft_consensus.cc:3037] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:28.541704 25392 raft_consensus.cc:504] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.542110 25392 leader_election.cc:288] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [CANDIDATE]: Term 1 election: Requested vote from peers ffe3cdf8471b4e1c958dbda73b75ed01 (127.23.50.130:35279), 2c0c9fb29b5c430b995e1f6e310951cd (127.23.50.131:35353)
I20260926 20:10:28.542479 25173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34518f01688946a5bf316b6d9a5d1e50" candidate_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01"
I20260926 20:10:28.542582 25173 raft_consensus.cc:3037] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:28.543458 25173 raft_consensus.cc:2454] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f94d5ca0d1cc40fda2fcd5ebd417ad91 in term 1.
I20260926 20:10:28.543717 25314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34518f01688946a5bf316b6d9a5d1e50" candidate_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0c9fb29b5c430b995e1f6e310951cd"
I20260926 20:10:28.543798 24964 leader_election.cc:302] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f94d5ca0d1cc40fda2fcd5ebd417ad91, ffe3cdf8471b4e1c958dbda73b75ed01; no voters:
I20260926 20:10:28.543848 25314 raft_consensus.cc:3037] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:28.543980 25392 raft_consensus.cc:2781] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:28.544205 25392 raft_consensus.cc:686] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 1 LEADER]: Becoming Leader. State: Replica: f94d5ca0d1cc40fda2fcd5ebd417ad91, State: Running, Role: LEADER
I20260926 20:10:28.544461 25392 consensus_queue.cc:237] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.545007 25314 raft_consensus.cc:2454] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f94d5ca0d1cc40fda2fcd5ebd417ad91 in term 1.
I20260926 20:10:28.547221 24873 catalog_manager.cc:5495] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f94d5ca0d1cc40fda2fcd5ebd417ad91 (127.23.50.129). New cstate: current_term: 1 leader_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:10:28 dist-test-slave-7034 krb5kdc[24828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453427, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:28.613543 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.603981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.23.50.190:46181)
0926 20:10:28.604131 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:28.604144 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:28.604370 (+ 226us) server_negotiation.cc:407] Connection header received
0926 20:10:28.604581 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:28.604585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:28.604624 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:28.604722 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:28.605442 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.606102 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.607017 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.607264 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.607384 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.610260 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:28.610263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:28.610265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:28.610288 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:28.611758 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:28.612276 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:28.612279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:28.612280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:28.612348 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:28.612628 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:28.612631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:28.612633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:28.612814 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:28.612919 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:28.613219 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 20:10:28.613411 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":50}
I20260926 20:10:28.613543 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:28.603930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:46181 (local address 127.0.0.1:48214)
0926 20:10:28.604259 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:28.604277 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:28.604418 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:28.604747 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:28.604750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:28.605210 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:28.605308 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.605315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.606239 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.606245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.606868 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:28.606877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:28.607391 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:28.607395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:28.607477 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:28.608165 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:28.608178 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:28.610087 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:28.611876 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:28.611888 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:28.611895 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:28.612132 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:28.612447 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:28.612450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:28.612452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:28.612533 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:28.612930 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:28.612939 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:28.613052 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 20:10:28.613172 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 20:10:28.613339 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":140,"threads_started":1}
I20260926 20:10:28.617759 24873 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:48214:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:10:28.618533 24873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:28.623389 25012 tablet_service.cc:1467] Processing CreateTablet for tablet 3761696d42f441528898c906aa1bf192 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=937f462c4e5c4c9ab9534ef5905261ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:10:28.623487 25152 tablet_service.cc:1467] Processing CreateTablet for tablet 3761696d42f441528898c906aa1bf192 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=937f462c4e5c4c9ab9534ef5905261ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:10:28.623633 25290 tablet_service.cc:1467] Processing CreateTablet for tablet 3761696d42f441528898c906aa1bf192 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=937f462c4e5c4c9ab9534ef5905261ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:10:28.623812 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3761696d42f441528898c906aa1bf192. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:28.623878 25152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3761696d42f441528898c906aa1bf192. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:28.623993 25290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3761696d42f441528898c906aa1bf192. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:28.626487 25385 tablet_bootstrap.cc:492] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01: Bootstrap starting.
I20260926 20:10:28.626694 25386 tablet_bootstrap.cc:492] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd: Bootstrap starting.
I20260926 20:10:28.626757 25384 tablet_bootstrap.cc:492] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: Bootstrap starting.
I20260926 20:10:28.627413 25385 tablet_bootstrap.cc:654] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:28.627563 25386 tablet_bootstrap.cc:654] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:28.627672 25384 tablet_bootstrap.cc:654] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:28.628665 25385 tablet_bootstrap.cc:492] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01: No bootstrap required, opened a new log
I20260926 20:10:28.628770 25385 ts_tablet_manager.cc:1397] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:28.629086 25384 tablet_bootstrap.cc:492] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: No bootstrap required, opened a new log
I20260926 20:10:28.629197 25384 ts_tablet_manager.cc:1397] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:28.629276 25385 raft_consensus.cc:348] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.629415 25385 raft_consensus.cc:374] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:28.629454 25385 raft_consensus.cc:729] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe3cdf8471b4e1c958dbda73b75ed01, State: Initialized, Role: FOLLOWER
I20260926 20:10:28.629516 25386 tablet_bootstrap.cc:492] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd: No bootstrap required, opened a new log
I20260926 20:10:28.629598 25386 ts_tablet_manager.cc:1397] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 20:10:28.629599 25385 consensus_queue.cc:260] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.629650 25384 raft_consensus.cc:348] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.629770 25384 raft_consensus.cc:374] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:28.629808 25384 raft_consensus.cc:729] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94d5ca0d1cc40fda2fcd5ebd417ad91, State: Initialized, Role: FOLLOWER
I20260926 20:10:28.629942 25385 ts_tablet_manager.cc:1428] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:10:28.629956 25384 consensus_queue.cc:260] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.630076 25386 raft_consensus.cc:348] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.630179 25386 raft_consensus.cc:374] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:28.630218 25386 raft_consensus.cc:729] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0c9fb29b5c430b995e1f6e310951cd, State: Initialized, Role: FOLLOWER
I20260926 20:10:28.630146 25391 tablet_replica.cc:439] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } }
I20260926 20:10:28.630282 25391 tablet_replica.cc:442] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:10:28.630213 25414 tablet_replica.cc:439] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } }
I20260926 20:10:28.630335 25414 tablet_replica.cc:442] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:10:28.630335 25386 consensus_queue.cc:260] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.630612 25386 ts_tablet_manager.cc:1428] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:10:28.630904 25384 ts_tablet_manager.cc:1428] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:10:28.630944 25390 tablet_replica.cc:439] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } }
I20260926 20:10:28.631060 25390 tablet_replica.cc:442] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 20:10:28.718089 25362 tablet.cc:2369] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:10:28.913148 25414 raft_consensus.cc:482] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:28.913365 25414 raft_consensus.cc:504] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.913913 25414 leader_election.cc:288] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe3cdf8471b4e1c958dbda73b75ed01 (127.23.50.130:35279), 2c0c9fb29b5c430b995e1f6e310951cd (127.23.50.131:35353)
I20260926 20:10:28.914386 25173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3761696d42f441528898c906aa1bf192" candidate_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" is_pre_election: true
I20260926 20:10:28.914418 25314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3761696d42f441528898c906aa1bf192" candidate_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" is_pre_election: true
I20260926 20:10:28.914542 25173 raft_consensus.cc:2454] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f94d5ca0d1cc40fda2fcd5ebd417ad91 in term 0.
I20260926 20:10:28.914608 25314 raft_consensus.cc:2454] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f94d5ca0d1cc40fda2fcd5ebd417ad91 in term 0.
I20260926 20:10:28.914877 24964 leader_election.cc:302] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f94d5ca0d1cc40fda2fcd5ebd417ad91, ffe3cdf8471b4e1c958dbda73b75ed01; no voters:
I20260926 20:10:28.915077 25414 raft_consensus.cc:2781] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:28.915136 25414 raft_consensus.cc:482] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:28.915167 25414 raft_consensus.cc:3037] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:28.967892 25414 raft_consensus.cc:504] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.968612 25414 leader_election.cc:288] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [CANDIDATE]: Term 1 election: Requested vote from peers ffe3cdf8471b4e1c958dbda73b75ed01 (127.23.50.130:35279), 2c0c9fb29b5c430b995e1f6e310951cd (127.23.50.131:35353)
I20260926 20:10:28.969245 25173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3761696d42f441528898c906aa1bf192" candidate_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01"
I20260926 20:10:28.969245 25314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3761696d42f441528898c906aa1bf192" candidate_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0c9fb29b5c430b995e1f6e310951cd"
I20260926 20:10:28.969400 25173 raft_consensus.cc:3037] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:28.969400 25314 raft_consensus.cc:3037] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:28.970723 25173 raft_consensus.cc:2454] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f94d5ca0d1cc40fda2fcd5ebd417ad91 in term 1.
I20260926 20:10:28.970723 25314 raft_consensus.cc:2454] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f94d5ca0d1cc40fda2fcd5ebd417ad91 in term 1.
I20260926 20:10:28.971139 24964 leader_election.cc:302] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f94d5ca0d1cc40fda2fcd5ebd417ad91, ffe3cdf8471b4e1c958dbda73b75ed01; no voters:
I20260926 20:10:28.971392 25414 raft_consensus.cc:2781] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:28.971508 25414 raft_consensus.cc:686] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 1 LEADER]: Becoming Leader. State: Replica: f94d5ca0d1cc40fda2fcd5ebd417ad91, State: Running, Role: LEADER
I20260926 20:10:28.971666 25414 consensus_queue.cc:237] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } }
I20260926 20:10:28.972468 25392 tablet_replica.cc:439] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: TxnStatusTablet state changed. Reason: New leader f94d5ca0d1cc40fda2fcd5ebd417ad91. Latest consensus state: current_term: 1 leader_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } }
I20260926 20:10:28.972577 25392 tablet_replica.cc:442] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:10:28.972808 25426 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:10:28.972982 25426 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:10:28.973768 24872 catalog_manager.cc:5495] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 reported cstate change: term changed from 0 to 1, leader changed from <none> to f94d5ca0d1cc40fda2fcd5ebd417ad91 (127.23.50.129). New cstate: current_term: 1 leader_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } health_report { overall_health: UNKNOWN } } }
I20260926 20:10:29.006695 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:29.001326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36189 (local address 127.0.0.1:53660)
0926 20:10:29.001494 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:29.001511 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:29.001626 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:29.002211 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:29.002214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:29.002224 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:10:29.002356 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:29.002362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.003542 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.003546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:29.004610 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:29.004624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.005273 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.005277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:29.005374 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:29.005442 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:10:29.006347 (+ 905us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:10:29.006368 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:10:29.006413 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 20:10:29.006522 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:10:29.007174 25429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:29.001621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.23.50.129:36189)
0926 20:10:29.001893 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:29.001898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:29.001912 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:10:29.001992 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:29.001995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:29.002058 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:29.002163 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:29.002508 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.003383 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.004749 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.005152 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.005249 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:29.005610 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:29.006226 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:29.006926 (+ 700us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:29.006962 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 20:10:29.007021 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260926 20:10:29.008288 25392 consensus_queue.cc:1035] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:29.011175 25313 raft_consensus.cc:1261] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd [term 1 FOLLOWER]: Refusing update from remote peer f94d5ca0d1cc40fda2fcd5ebd417ad91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:29.011296 25173 raft_consensus.cc:1261] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 1 FOLLOWER]: Refusing update from remote peer f94d5ca0d1cc40fda2fcd5ebd417ad91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:29.011718 25414 consensus_queue.cc:1035] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:29.012539 25414 consensus_queue.cc:1035] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:29.012959 25391 tablet_replica.cc:439] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01: TxnStatusTablet state changed. Reason: New leader f94d5ca0d1cc40fda2fcd5ebd417ad91. Latest consensus state: current_term: 1 leader_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } }
I20260926 20:10:29.013070 25391 tablet_replica.cc:442] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:10:29.014487 25390 tablet_replica.cc:439] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd: TxnStatusTablet state changed. Reason: New leader f94d5ca0d1cc40fda2fcd5ebd417ad91. Latest consensus state: current_term: 1 leader_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } }
I20260926 20:10:29.014602 25390 tablet_replica.cc:442] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:10:29.015774 25392 tablet_replica.cc:439] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } }
I20260926 20:10:29.015875 25392 tablet_replica.cc:442] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:10:29.016391 25414 tablet_replica.cc:439] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } }
I20260926 20:10:29.016494 25414 tablet_replica.cc:442] T 3761696d42f441528898c906aa1bf192 P f94d5ca0d1cc40fda2fcd5ebd417ad91: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:10:29.017302 25391 tablet_replica.cc:439] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } }
I20260926 20:10:29.017412 25391 tablet_replica.cc:442] T 3761696d42f441528898c906aa1bf192 P ffe3cdf8471b4e1c958dbda73b75ed01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:10:29.018394 25390 tablet_replica.cc:439] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f94d5ca0d1cc40fda2fcd5ebd417ad91" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36189 } } peers { permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 } } peers { permanent_uuid: "2c0c9fb29b5c430b995e1f6e310951cd" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 35353 } } }
I20260926 20:10:29.018556 25390 tablet_replica.cc:442] T 3761696d42f441528898c906aa1bf192 P 2c0c9fb29b5c430b995e1f6e310951cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:10:29.020151 25392 consensus_queue.cc:1035] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe3cdf8471b4e1c958dbda73b75ed01" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 35279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:29.027101 25429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:29.023617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.23.50.129:36189)
0926 20:10:29.023775 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:29.023781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:29.023796 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:10:29.023839 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:29.023842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:29.023887 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:29.023953 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:29.024260 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.024958 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.025950 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.026259 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.026380 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:29.026659 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:29.026736 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:29.026782 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:29.026951 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 20:10:29.027008 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:10:29.037896 25429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:29.034611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:43475 (local address 127.23.50.129:36189)
0926 20:10:29.034725 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:29.034731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:29.034851 (+ 120us) server_negotiation.cc:407] Connection header received
0926 20:10:29.034997 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:29.035002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:29.035041 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:29.035126 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:29.035458 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.035947 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.036786 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.037006 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.037112 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:29.037409 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:29.037481 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:29.037560 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:29.037644 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 20:10:29.037716 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":49}
I20260926 20:10:29.037905 25445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:29.034497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36189 (local address 127.23.50.129:43475)
0926 20:10:29.034766 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:29.034780 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:29.034891 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:29.035132 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:29.035134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:29.035141 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:10:29.035348 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:29.035355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.036048 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.036054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:29.036670 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:29.036677 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.037122 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.037125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:29.037240 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:29.037278 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:10:29.037568 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:10:29.037575 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:10:29.037637 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 20:10:29.037740 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":62,"thread_start_us":88,"threads_started":1}
I20260926 20:10:29.085145 24873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:29.085295 24873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:29.087512 24873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0489d5334224cbda92584143b5cf3ea: Sending DeleteTablet for 3 replicas of tablet 34518f01688946a5bf316b6d9a5d1e50
I20260926 20:10:29.088150 25152 tablet_service.cc:1514] Processing DeleteTablet for tablet 34518f01688946a5bf316b6d9a5d1e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:29 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:51320
I20260926 20:10:29.088127 25290 tablet_service.cc:1514] Processing DeleteTablet for tablet 34518f01688946a5bf316b6d9a5d1e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:29 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:38584
I20260926 20:10:29.088467 25447 tablet_replica.cc:330] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01: stopping tablet replica
I20260926 20:10:29.088593 25447 raft_consensus.cc:2229] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:29.088712 25447 raft_consensus.cc:2258] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:29.088896 25011 tablet_service.cc:1514] Processing DeleteTablet for tablet 34518f01688946a5bf316b6d9a5d1e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:29 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:53654
I20260926 20:10:29.089198 25450 tablet_replica.cc:330] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91: stopping tablet replica
I20260926 20:10:29.089272 25448 tablet_replica.cc:330] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd: stopping tablet replica
I20260926 20:10:29.089408 25448 raft_consensus.cc:2229] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:29.089409 25450 raft_consensus.cc:2229] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:10:29.089534 25448 raft_consensus.cc:2258] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:29.089723 25447 ts_tablet_manager.cc:1905] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:29.089931 25450 raft_consensus.cc:2258] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:29.090749 25450 ts_tablet_manager.cc:1905] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:29.090912 25448 ts_tablet_manager.cc:1905] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:29.092710 25447 ts_tablet_manager.cc:1918] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:10:29.092794 25447 log.cc:1196] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/34518f01688946a5bf316b6d9a5d1e50
I20260926 20:10:29.093068 25447 ts_tablet_manager.cc:1939] T 34518f01688946a5bf316b6d9a5d1e50 P ffe3cdf8471b4e1c958dbda73b75ed01: Deleting consensus metadata
I20260926 20:10:29.093611 25448 ts_tablet_manager.cc:1918] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:10:29.093701 25448 log.cc:1196] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/34518f01688946a5bf316b6d9a5d1e50
I20260926 20:10:29.093787 24859 catalog_manager.cc:4841] TS ffe3cdf8471b4e1c958dbda73b75ed01 (127.23.50.130:35279): tablet 34518f01688946a5bf316b6d9a5d1e50 (table test-table [id=afce46b24adf4d76a0ac1df856c72194]) successfully deleted
I20260926 20:10:29.093988 25448 ts_tablet_manager.cc:1939] T 34518f01688946a5bf316b6d9a5d1e50 P 2c0c9fb29b5c430b995e1f6e310951cd: Deleting consensus metadata
I20260926 20:10:29.094079 25450 ts_tablet_manager.cc:1918] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:10:29.094161 25450 log.cc:1196] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790453421225356-23754-0/minicluster-data/ts-0/wal/wals/34518f01688946a5bf316b6d9a5d1e50
I20260926 20:10:29.094429 25450 ts_tablet_manager.cc:1939] T 34518f01688946a5bf316b6d9a5d1e50 P f94d5ca0d1cc40fda2fcd5ebd417ad91: Deleting consensus metadata
I20260926 20:10:29.094734 24857 catalog_manager.cc:4841] TS 2c0c9fb29b5c430b995e1f6e310951cd (127.23.50.131:35353): tablet 34518f01688946a5bf316b6d9a5d1e50 (table test-table [id=afce46b24adf4d76a0ac1df856c72194]) successfully deleted
I20260926 20:10:29.095151 24857 catalog_manager.cc:4841] TS f94d5ca0d1cc40fda2fcd5ebd417ad91 (127.23.50.129:36189): tablet 34518f01688946a5bf316b6d9a5d1e50 (table test-table [id=afce46b24adf4d76a0ac1df856c72194]) successfully deleted
Sep 26 20:10:29 dist-test-slave-7034 krb5kdc[24828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453428, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.50.129@KRBTEST.COM
I20260926 20:10:29.106053 25429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:29.089985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.23.50.129:36189)
0926 20:10:29.090101 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:29.090106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:29.095029 (+ 4923us) server_negotiation.cc:407] Connection header received
0926 20:10:29.095200 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:29.095203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:29.095255 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:29.095342 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:29.096296 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.096813 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.098045 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.098258 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.098337 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:29.102247 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:29.102262 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:29.102273 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:29.102296 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:29.104187 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:29.104637 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:29.104643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:29.104644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:29.104734 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:29.105074 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:29.105076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:29.105077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:29.105319 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:29.105461 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:29.105705 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 20:10:29.105811 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 20:10:29.106499 25072 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53686
I20260926 20:10:29.115947 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:29.108275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.23.50.190:46181)
0926 20:10:29.108413 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:29.108418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:29.108466 (+ 48us) server_negotiation.cc:407] Connection header received
0926 20:10:29.108570 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:29.108573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:29.108619 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:29.108703 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:29.109283 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.110121 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.110935 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:29.111257 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:29.111366 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:29.112664 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:29.112668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:29.112671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:29.112698 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:29.114330 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:29.114727 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:29.114733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:29.114735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:29.114821 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:29.115063 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:29.115066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:29.115068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:29.115287 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:29.115422 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:29.115616 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 20:10:29.115785 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 20:10:29.116391 24873 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48226
I20260926 20:10:29.117722 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 24946
I20260926 20:10:29.125226 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 25087
I20260926 20:10:29.131335 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 25228
I20260926 20:10:29.138059 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 24841
2026-09-26T20:10:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3012 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:10:29 dist-test-slave-7034 krb5kdc[25456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:10:29 dist-test-slave-7034 krb5kdc[25456](info): set up 2 sockets
Sep 26 20:10:29 dist-test-slave-7034 krb5kdc[25456](info): commencing operation
krb5kdc: starting...
W20260926 20:10:30.178763 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:10:30 dist-test-slave-7034 krb5kdc[25456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:10:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:10:30.307081 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:33455
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:30.419740 25470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:30.420007 25470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:30.420047 25470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:30.423941 25470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:30.423988 25470 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:10:30.424005 25470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:30.424023 25470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:30.424039 25470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:30.424054 25470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:30.429314 25470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:33455
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:33455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:30.430425 25470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:30.431384 25470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:10:30.440227 25470 server_base.cc:1034] running on GCE node
W20260926 20:10:30.440187 25475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:30.440254 25476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:30.440179 25478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:30.441000 25470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:30.442231 25470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:30.443449 25470 hybrid_clock.cc:648] HybridClock initialized: now 1790453430443398 us; error 63 us; skew 500 ppm
Sep 26 20:10:30 dist-test-slave-7034 krb5kdc[25456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453430, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:30.446467 25470 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:10:30.447309 25470 webserver.cc:460] Webserver started at http://127.23.50.190:33899/ using document root <none> and password file <none>
I20260926 20:10:30.447830 25470 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:30.447886 25470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:30.448047 25470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:30.450125 25470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "c05b50859cf64b4fb7e2ceb9fe4c8070"
format_stamp: "Formatted at 2026-09-26 20:10:30 on dist-test-slave-7034"
server_key: "93a868c9aec8f1162eabfa6b6dc5bba1"
server_key_iv: "1bac3d9b88078e97c2d790a8549b5055"
server_key_version: "encryptionkey@0"
I20260926 20:10:30.450623 25470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "c05b50859cf64b4fb7e2ceb9fe4c8070"
format_stamp: "Formatted at 2026-09-26 20:10:30 on dist-test-slave-7034"
server_key: "93a868c9aec8f1162eabfa6b6dc5bba1"
server_key_iv: "1bac3d9b88078e97c2d790a8549b5055"
server_key_version: "encryptionkey@0"
I20260926 20:10:30.454298 25470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:30.456683 25485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:30.458000 25470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:10:30.458119 25470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "c05b50859cf64b4fb7e2ceb9fe4c8070"
format_stamp: "Formatted at 2026-09-26 20:10:30 on dist-test-slave-7034"
server_key: "93a868c9aec8f1162eabfa6b6dc5bba1"
server_key_iv: "1bac3d9b88078e97c2d790a8549b5055"
server_key_version: "encryptionkey@0"
I20260926 20:10:30.458232 25470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:30.467710 25470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:30.471009 25470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:30.471213 25470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:30.478890 25470 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:33455
I20260926 20:10:30.478916 25547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:33455 every 8 connection(s)
I20260926 20:10:30.479925 25470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:30.482811 25548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:30.483716 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 25470
I20260926 20:10:30.483846 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:30.484117 23754 external_mini_cluster.cc:1428] Setting key b98242e384e2db3c0481d04147ef918b
I20260926 20:10:30.489485 25548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070: Bootstrap starting.
I20260926 20:10:30.492331 25548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:30.493201 25548 log.cc:824] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:10:30 dist-test-slave-7034 krb5kdc[25456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:30.495298 25548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070: No bootstrap required, opened a new log
I20260926 20:10:30.498915 25548 raft_consensus.cc:348] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05b50859cf64b4fb7e2ceb9fe4c8070" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 33455 } }
I20260926 20:10:30.499182 25548 raft_consensus.cc:374] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:30.499269 25548 raft_consensus.cc:729] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05b50859cf64b4fb7e2ceb9fe4c8070, State: Initialized, Role: FOLLOWER
I20260926 20:10:30.499715 25548 consensus_queue.cc:260] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05b50859cf64b4fb7e2ceb9fe4c8070" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 33455 } }
I20260926 20:10:30.499871 25548 raft_consensus.cc:388] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:30.499958 25548 raft_consensus.cc:482] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:30.500058 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:30.485591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39776 (local address 127.23.50.190:33455)
0926 20:10:30.486044 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:30.486060 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:30.486096 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:10:30.486736 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:30.486754 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:30.487121 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:30.487412 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:30.488344 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.489337 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.490528 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.491014 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.491144 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:30.494112 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:30.494125 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:30.494141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:30.494173 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:30.496537 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:30.497081 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:30.497087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:30.497091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:30.497231 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:30.497577 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:30.497580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:30.497581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:30.498128 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:30.498326 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:30.498611 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 20:10:30.498989 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":144,"threads_started":1}
I20260926 20:10:30.500075 25548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:30.501061 25548 raft_consensus.cc:504] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05b50859cf64b4fb7e2ceb9fe4c8070" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 33455 } }
I20260926 20:10:30.501430 25548 leader_election.cc:302] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c05b50859cf64b4fb7e2ceb9fe4c8070; no voters:
I20260926 20:10:30.501787 25548 leader_election.cc:288] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:30.501931 25553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:30.502161 25553 raft_consensus.cc:686] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [term 1 LEADER]: Becoming Leader. State: Replica: c05b50859cf64b4fb7e2ceb9fe4c8070, State: Running, Role: LEADER
I20260926 20:10:30.502571 25553 consensus_queue.cc:237] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05b50859cf64b4fb7e2ceb9fe4c8070" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 33455 } }
I20260926 20:10:30.503129 25548 sys_catalog.cc:564] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:30.504349 25555 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c05b50859cf64b4fb7e2ceb9fe4c8070. Latest consensus state: current_term: 1 leader_uuid: "c05b50859cf64b4fb7e2ceb9fe4c8070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05b50859cf64b4fb7e2ceb9fe4c8070" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 33455 } } }
I20260926 20:10:30.504480 25555 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:30.504850 25562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:30.504801 25554 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c05b50859cf64b4fb7e2ceb9fe4c8070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c05b50859cf64b4fb7e2ceb9fe4c8070" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 33455 } } }
I20260926 20:10:30.504966 25554 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:30.508211 25562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:30.514315 25562 catalog_manager.cc:1347] Generated new cluster ID: e8bc1e6b51434c62a9afa2f21686d670
I20260926 20:10:30.514397 25562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:30.532559 25562 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:30.533950 25562 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:30.538928 25562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070: Generated new TSK 0
I20260926 20:10:30.539846 25562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:10:30.572790 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:33455
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:30.685472 25574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:30.685736 25574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:30.685784 25574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:30.690013 25574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:30.690073 25574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:30.690114 25574 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:10:30.690129 25574 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:10:30.690163 25574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:10:30.695498 25574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.50.190:33455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:30.696484 25574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:30.697440 25574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:30.707185 25579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:30.707178 25580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:30.707301 25574 server_base.cc:1034] running on GCE node
W20260926 20:10:30.707178 25582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:30.707823 25574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:30.708483 25574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:30.709685 25574 hybrid_clock.cc:648] HybridClock initialized: now 1790453430709649 us; error 46 us; skew 500 ppm
Sep 26 20:10:30 dist-test-slave-7034 krb5kdc[25456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453430, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:30.712602 25574 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:10:30.713533 25574 webserver.cc:460] Webserver started at http://127.23.50.129:44739/ using document root <none> and password file <none>
I20260926 20:10:30.714176 25574 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:30.714268 25574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:30.714509 25574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:30.716542 25574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "2427ff8243d44f7099b5d1d94db18d36"
format_stamp: "Formatted at 2026-09-26 20:10:30 on dist-test-slave-7034"
server_key: "107a5df86e4ccd9ede055ee16a74f327"
server_key_iv: "d6b83391bf9196ecbb0b680ebdbc5d9e"
server_key_version: "encryptionkey@0"
I20260926 20:10:30.717118 25574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "2427ff8243d44f7099b5d1d94db18d36"
format_stamp: "Formatted at 2026-09-26 20:10:30 on dist-test-slave-7034"
server_key: "107a5df86e4ccd9ede055ee16a74f327"
server_key_iv: "d6b83391bf9196ecbb0b680ebdbc5d9e"
server_key_version: "encryptionkey@0"
I20260926 20:10:30.720991 25574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:10:30.723675 25589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:30.725000 25574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:30.725157 25574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "2427ff8243d44f7099b5d1d94db18d36"
format_stamp: "Formatted at 2026-09-26 20:10:30 on dist-test-slave-7034"
server_key: "107a5df86e4ccd9ede055ee16a74f327"
server_key_iv: "d6b83391bf9196ecbb0b680ebdbc5d9e"
server_key_version: "encryptionkey@0"
I20260926 20:10:30.725289 25574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:30.755276 25574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:30.758440 25574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:30.758652 25574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:30.760103 25574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:30.760182 25574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:30.760255 25574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:30.760306 25574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:10:30 dist-test-slave-7034 krb5kdc[25456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453430, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:30.771049 25574 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:37173
I20260926 20:10:30.771114 25705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:37173 every 8 connection(s)
I20260926 20:10:30.771999 25574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:10:30.774904 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:30.761182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:59689 (local address 127.23.50.190:33455)
0926 20:10:30.761334 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:30.761338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:30.762426 (+ 1088us) server_negotiation.cc:407] Connection header received
0926 20:10:30.763827 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:30.763833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:30.763903 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:30.764009 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:30.765625 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.766575 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.767892 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.768214 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.768331 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:30.771259 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:30.771267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:30.771269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:30.771293 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:30.772881 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:30.773375 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:30.773380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:30.773382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:30.773463 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:30.773804 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:30.773808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:30.773810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:30.774016 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:30.774120 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:30.774608 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 20:10:30.774733 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:10:30.775642 25598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:30.761721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:33455 (local address 127.23.50.129:59689)
0926 20:10:30.762251 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:30.762296 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:30.763504 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:30.764218 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:30.764229 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:30.764727 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:30.765441 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:30.765462 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.766726 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.766730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:30.767761 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:30.767771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.768369 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.768373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:30.768557 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:30.769198 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:30.769222 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:30.771074 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:30.772994 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:30.772999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:30.773013 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:30.773269 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:30.773555 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:30.773560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:30.773562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:30.773707 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:30.774131 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:30.774139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:30.774371 (+ 232us) client_negotiation.cc:769] Sending connection context
0926 20:10:30.774556 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 20:10:30.774868 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":134,"threads_started":1}
I20260926 20:10:30.777081 25706 heartbeater.cc:347] Connected to a master server at 127.23.50.190:33455
I20260926 20:10:30.777263 25706 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:30.777804 25706 heartbeater.cc:511] Master 127.23.50.190:33455 requested a full tablet report, sending...
I20260926 20:10:30.779506 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 25574
I20260926 20:10:30.779625 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:10:30.779600 25501 ts_manager.cc:194] Registered new tserver with Master: 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173)
I20260926 20:10:30.779940 23754 external_mini_cluster.cc:1428] Setting key 3a5077d24466e7b4f42f74cb405ed90d
I20260926 20:10:30.781168 25501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:59689
I20260926 20:10:30.787222 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:30.779331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:34109 (local address 127.23.50.190:33455)
0926 20:10:30.779448 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:30.779451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:30.779673 (+ 222us) server_negotiation.cc:407] Connection header received
0926 20:10:30.779854 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:30.779856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:30.779894 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:30.779976 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:30.780437 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.781216 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.782336 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.782568 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.782679 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:30.784097 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:30.784099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:30.784101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:30.784123 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:30.785530 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:30.786042 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:30.786046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:30.786048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:30.786131 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:30.786420 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:30.786424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:30.786426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:30.786595 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:30.786699 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:30.786953 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 20:10:30.787057 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":48}
I20260926 20:10:30.787228 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:30.779167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:33455 (local address 127.23.50.129:34109)
0926 20:10:30.779594 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:30.779608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:30.779743 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:30.779991 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:30.779994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:30.780191 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:30.780327 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:30.780335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.781371 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.781375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:30.782219 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:30.782227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.782681 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.782685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:30.782787 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:30.783443 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:30.783459 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:30.783942 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:30.785642 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:30.785647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:30.785649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:30.785920 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:30.786240 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:30.786243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:30.786245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:30.786330 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:30.786695 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:30.786699 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:30.786793 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 20:10:30.786899 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 20:10:30.787036 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:10:30.808853 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:33455
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:30.921149 25715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:30.921440 25715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:30.921490 25715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:30.925696 25715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:30.925768 25715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:30.925812 25715 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:10:30.925825 25715 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:10:30.925925 25715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:10:30.931488 25715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.50.190:33455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:30.932616 25715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:30.933616 25715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:30.944029 25721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:30.944142 25715 server_base.cc:1034] running on GCE node
W20260926 20:10:30.944029 25723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:30.944105 25720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:30.944877 25715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:30.945477 25715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:30.946642 25715 hybrid_clock.cc:648] HybridClock initialized: now 1790453430946628 us; error 45 us; skew 500 ppm
Sep 26 20:10:30 dist-test-slave-7034 krb5kdc[25456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453430, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:30.949453 25715 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:10:30.950440 25715 webserver.cc:460] Webserver started at http://127.23.50.130:39445/ using document root <none> and password file <none>
I20260926 20:10:30.951016 25715 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:30.951076 25715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:30.951341 25715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:30.953375 25715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "8648836dffb143cbb04148defddfed80"
format_stamp: "Formatted at 2026-09-26 20:10:30 on dist-test-slave-7034"
server_key: "3de41c6aa9dbfaa438dd83a2b0e3be3d"
server_key_iv: "f9af560953420067f0665adfc989cd40"
server_key_version: "encryptionkey@0"
I20260926 20:10:30.954039 25715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "8648836dffb143cbb04148defddfed80"
format_stamp: "Formatted at 2026-09-26 20:10:30 on dist-test-slave-7034"
server_key: "3de41c6aa9dbfaa438dd83a2b0e3be3d"
server_key_iv: "f9af560953420067f0665adfc989cd40"
server_key_version: "encryptionkey@0"
I20260926 20:10:30.957958 25715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:10:30.960376 25730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:30.961817 25715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:30.962038 25715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "8648836dffb143cbb04148defddfed80"
format_stamp: "Formatted at 2026-09-26 20:10:30 on dist-test-slave-7034"
server_key: "3de41c6aa9dbfaa438dd83a2b0e3be3d"
server_key_iv: "f9af560953420067f0665adfc989cd40"
server_key_version: "encryptionkey@0"
I20260926 20:10:30.962191 25715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:30.977559 25715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:30.980890 25715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:30.981117 25715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:30.982684 25715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:30.982767 25715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:30.982842 25715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:30.982894 25715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:10:30 dist-test-slave-7034 krb5kdc[25456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453430, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:30.993968 25715 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:38489
I20260926 20:10:30.994419 25846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:38489 every 8 connection(s)
I20260926 20:10:30.995143 25715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:10:30.998198 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:30.983544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:34345 (local address 127.23.50.190:33455)
0926 20:10:30.983684 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:30.983688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:30.984765 (+ 1077us) server_negotiation.cc:407] Connection header received
0926 20:10:30.986030 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:30.986033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:30.986097 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:30.986263 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:30.987754 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.988381 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.989658 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.990121 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.990267 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:30.994372 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:30.994376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:30.994379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:30.994414 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:30.996089 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:30.996615 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:30.996619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:30.996620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:30.996712 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:30.997065 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:30.997069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:30.997071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:30.997283 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:30.997393 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:30.997913 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 20:10:30.998037 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:10:30.999030 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:30.984081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:33455 (local address 127.23.50.130:34345)
0926 20:10:30.984593 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:30.984638 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:30.985736 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:30.986448 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:30.986455 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:30.986903 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:30.987579 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:30.987596 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.988524 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.988529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:30.989501 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:30.989511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:30.990328 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:30.990336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:30.990583 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:30.992018 (+ 1435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:30.992042 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:30.994097 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:30.996215 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:30.996221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:30.996236 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:30.996498 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:30.996822 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:30.996825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:30.996827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:30.996968 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:30.997388 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:30.997395 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:30.997644 (+ 249us) client_negotiation.cc:769] Sending connection context
0926 20:10:30.997845 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 20:10:30.998198 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":149,"threads_started":1}
I20260926 20:10:31.000474 25847 heartbeater.cc:347] Connected to a master server at 127.23.50.190:33455
I20260926 20:10:31.000671 25847 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:31.001302 25847 heartbeater.cc:511] Master 127.23.50.190:33455 requested a full tablet report, sending...
I20260926 20:10:31.002517 25501 ts_manager.cc:194] Registered new tserver with Master: 8648836dffb143cbb04148defddfed80 (127.23.50.130:38489)
I20260926 20:10:31.003271 25501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:34345
I20260926 20:10:31.005353 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 25715
I20260926 20:10:31.005487 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:10:31.005787 23754 external_mini_cluster.cc:1428] Setting key 17ce364083f1d08e12f7a9889ac99417
I20260926 20:10:31.011490 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.002642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:58429 (local address 127.23.50.190:33455)
0926 20:10:31.003571 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.003575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.003589 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:10:31.003623 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.003627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.003665 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.003748 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:31.004267 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.004998 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.006106 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.006347 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.006462 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.007758 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:31.007762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:31.007765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:31.007794 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:31.009651 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:31.010203 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:31.010206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:31.010207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:31.010290 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:31.010630 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:31.010633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:31.010634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:31.010853 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:31.010941 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.011260 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.011368 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862}
I20260926 20:10:31.011641 25853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.002586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:33455 (local address 127.23.50.130:58429)
0926 20:10:31.002873 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.002888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.002997 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.003774 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.003778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.004014 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:31.004132 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.004139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.005157 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.005160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.005993 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.006002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.006579 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.006583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.006662 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.007301 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:31.007312 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:31.007609 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:31.009780 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:31.009785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:31.009787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:31.010091 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:31.010396 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:31.010400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:31.010404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:31.010507 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:31.010958 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:31.010964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:31.011081 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.011198 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.011358 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:10:31.032998 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:33455
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:31.157868 25856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:31.158200 25856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:31.158306 25856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:31.162542 25856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:31.162601 25856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:31.162642 25856 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 20:10:31.162657 25856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:10:31.162688 25856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:10:31.168171 25856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.50.190:33455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:31.169296 25856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:31.170282 25856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:31.180008 25862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:31.180002 25861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:31.180002 25864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:31.180104 25856 server_base.cc:1034] running on GCE node
I20260926 20:10:31.180678 25856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:31.181375 25856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:31.182580 25856 hybrid_clock.cc:648] HybridClock initialized: now 1790453431182485 us; error 106 us; skew 500 ppm
Sep 26 20:10:31 dist-test-slave-7034 krb5kdc[25456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453431, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:31.185662 25856 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:10:31.186671 25856 webserver.cc:460] Webserver started at http://127.23.50.131:46495/ using document root <none> and password file <none>
I20260926 20:10:31.187300 25856 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:31.187405 25856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:31.187674 25856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:31.190057 25856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "4a522790944e41e4bece9ca987e64bf5"
format_stamp: "Formatted at 2026-09-26 20:10:31 on dist-test-slave-7034"
server_key: "361885cd0578039a6b038512732908c8"
server_key_iv: "0bf8fa49e9904c248ddb1bac278bafb2"
server_key_version: "encryptionkey@0"
I20260926 20:10:31.190688 25856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "4a522790944e41e4bece9ca987e64bf5"
format_stamp: "Formatted at 2026-09-26 20:10:31 on dist-test-slave-7034"
server_key: "361885cd0578039a6b038512732908c8"
server_key_iv: "0bf8fa49e9904c248ddb1bac278bafb2"
server_key_version: "encryptionkey@0"
I20260926 20:10:31.194749 25856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:10:31.197232 25871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:31.198493 25856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 20:10:31.198634 25856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "4a522790944e41e4bece9ca987e64bf5"
format_stamp: "Formatted at 2026-09-26 20:10:31 on dist-test-slave-7034"
server_key: "361885cd0578039a6b038512732908c8"
server_key_iv: "0bf8fa49e9904c248ddb1bac278bafb2"
server_key_version: "encryptionkey@0"
I20260926 20:10:31.198756 25856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:31.224679 25856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:31.228325 25856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:31.228538 25856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:31.230270 25856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:31.230322 25856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:31.230363 25856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:31.230414 25856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:31.241658 25856 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:37943
Sep 26 20:10:31 dist-test-slave-7034 krb5kdc[25456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453431, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:31.242921 25856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:10:31.244022 25986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:37943 every 8 connection(s)
I20260926 20:10:31.247342 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.231279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:49817 (local address 127.23.50.190:33455)
0926 20:10:31.231442 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.231447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.232329 (+ 882us) server_negotiation.cc:407] Connection header received
0926 20:10:31.233505 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.233508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.233579 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.233692 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:31.235343 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.236265 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.238303 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.238688 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.238810 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.242984 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:31.242988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:31.242991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:31.243024 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:31.244892 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:31.245660 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:31.245664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:31.245666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:31.245763 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:31.246276 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:31.246280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:31.246282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:31.246496 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:31.246581 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.247087 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.247219 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:10:31.248176 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.231684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:33455 (local address 127.23.50.131:49817)
0926 20:10:31.232160 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.232212 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.233258 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.233926 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.233936 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.234421 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:31.235163 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.235182 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.236449 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.236453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.238104 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.238119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.238938 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.238946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.239340 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.240248 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:31.240277 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:31.242696 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:31.245087 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:31.245096 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:31.245134 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:31.245530 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:31.245884 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:31.245887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:31.245889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:31.246086 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:31.246594 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:31.246599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:31.246861 (+ 262us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.247019 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.247300 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":135,"threads_started":1}
I20260926 20:10:31.249404 25987 heartbeater.cc:347] Connected to a master server at 127.23.50.190:33455
I20260926 20:10:31.249547 25987 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:31.250170 25987 heartbeater.cc:511] Master 127.23.50.190:33455 requested a full tablet report, sending...
I20260926 20:10:31.251065 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 25856
I20260926 20:10:31.251192 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:10:31.251394 25502 ts_manager.cc:194] Registered new tserver with Master: 4a522790944e41e4bece9ca987e64bf5 (127.23.50.131:37943)
I20260926 20:10:31.251505 23754 external_mini_cluster.cc:1428] Setting key 1c32afe72f5229b04129af38590322e2
I20260926 20:10:31.252195 25502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:49817
I20260926 20:10:31.254036 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:10:31.263864 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.251986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:54817 (local address 127.23.50.190:33455)
0926 20:10:31.252155 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.252160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.252707 (+ 547us) server_negotiation.cc:407] Connection header received
0926 20:10:31.255868 (+ 3161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.255874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.255932 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.256033 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:31.256488 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.257229 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.258654 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.258886 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.258978 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.260295 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:31.260297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:31.260299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:31.260322 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:31.261935 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:31.262441 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:31.262444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:31.262445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:31.262522 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:31.262866 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:31.262868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:31.262870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:31.263175 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:31.263302 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.263629 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.263741 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 20:10:31.264058 25993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.251883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:33455 (local address 127.23.50.131:54817)
0926 20:10:31.252588 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.252603 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.253902 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.256058 (+ 2156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.256060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.256284 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:31.256379 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.256385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.257405 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.257412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.258495 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.258510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.259022 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.259026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.259140 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.259756 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:31.259768 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:31.260095 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:31.262059 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:31.262066 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:31.262071 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:31.262348 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:31.262615 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:31.262621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:31.262626 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:31.262738 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:31.263312 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:31.263319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:31.263447 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.263574 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.263756 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":99,"threads_started":1}
Sep 26 20:10:31 dist-test-slave-7034 krb5kdc[25456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:10:31 dist-test-slave-7034 krb5kdc[25456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:31.281728 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.273387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39778 (local address 127.23.50.190:33455)
0926 20:10:31.273506 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.273513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.273526 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:10:31.273564 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.273566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.273607 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.273693 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:31.274440 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.274980 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.275805 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.276068 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.276188 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.278666 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:31.278669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:31.278672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:31.278696 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:31.280171 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:31.280554 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:31.280557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:31.280558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:31.280633 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:31.280875 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:31.280877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:31.280878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:31.281044 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:31.281158 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.281397 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.281494 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:10:31.284724 25501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:10:31.287791 25501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:31.302582 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.296255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:37173 (local address 127.0.0.1:34968)
0926 20:10:31.297114 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.297167 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.297289 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.298277 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.298279 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.298296 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:31.298413 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.298420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.299535 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.299541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.300802 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.300813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.301964 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.301968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.302091 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.302136 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.302264 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.302388 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":76,"thread_start_us":91,"threads_started":1}
I20260926 20:10:31.302527 26005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.296407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34968 (local address 127.23.50.129:37173)
0926 20:10:31.297887 (+ 1480us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.297893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.297913 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:10:31.297979 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.297983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.298137 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.298574 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:31.298686 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.299380 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.300925 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.301832 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.301942 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.302097 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.302296 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.302382 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1361,"thread_start_us":73,"threads_started":1}
I20260926 20:10:31.303543 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.296580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:38489 (local address 127.0.0.1:59140)
0926 20:10:31.297556 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.297570 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.297666 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.298359 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.298361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.298378 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:31.298463 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.298472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.299802 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.299805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.301489 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.301498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.302727 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.303161 (+ 434us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.303260 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.303283 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.303346 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.303412 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":775,"thread_start_us":81,"threads_started":1}
I20260926 20:10:31.304100 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.297104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.23.50.130:38489)
0926 20:10:31.297607 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.297621 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.297647 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:10:31.297800 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.297805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.298041 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.298213 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:31.298579 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.299680 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.301613 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.302478 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.302635 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.303522 (+ 887us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.303567 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.303967 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":111,"threads_started":1}
I20260926 20:10:31.304180 26007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.296484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:37943 (local address 127.0.0.1:33300)
0926 20:10:31.297508 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.297522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.297600 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.298137 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.298142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.298164 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:31.298299 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.298305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.299758 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.299761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.300818 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.300828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.303692 (+ 2864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.303696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.303841 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.303864 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.303962 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.304059 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":50,"thread_start_us":108,"threads_started":1}
I20260926 20:10:31.304891 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.297160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.23.50.131:37943)
0926 20:10:31.297580 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.297596 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.297621 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:10:31.297700 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.297707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.297979 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.298136 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:31.298410 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.299617 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.300972 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.301742 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.301943 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.302128 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.303999 (+ 1871us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.304101 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":108,"threads_started":1}
I20260926 20:10:31.306150 25640 tablet_service.cc:1467] Processing CreateTablet for tablet 300b545cce99486aa8aa8d386186c0b4 (DEFAULT_TABLE table=test-table [id=27995b6691484b7e96f0c0789fdc5af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:31.306680 25781 tablet_service.cc:1467] Processing CreateTablet for tablet 300b545cce99486aa8aa8d386186c0b4 (DEFAULT_TABLE table=test-table [id=27995b6691484b7e96f0c0789fdc5af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:31.307276 25640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300b545cce99486aa8aa8d386186c0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:31.307663 25781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300b545cce99486aa8aa8d386186c0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:31.307591 25921 tablet_service.cc:1467] Processing CreateTablet for tablet 300b545cce99486aa8aa8d386186c0b4 (DEFAULT_TABLE table=test-table [id=27995b6691484b7e96f0c0789fdc5af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:31.308552 25921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 300b545cce99486aa8aa8d386186c0b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:31.313305 26010 tablet_bootstrap.cc:492] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36: Bootstrap starting.
I20260926 20:10:31.315348 26011 tablet_bootstrap.cc:492] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80: Bootstrap starting.
I20260926 20:10:31.315668 26010 tablet_bootstrap.cc:654] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:31.316425 26012 tablet_bootstrap.cc:492] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5: Bootstrap starting.
I20260926 20:10:31.316623 26010 log.cc:824] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:31.317495 26011 tablet_bootstrap.cc:654] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:31.318363 26012 tablet_bootstrap.cc:654] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:31.318346 26011 log.cc:824] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:31.318885 26010 tablet_bootstrap.cc:492] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36: No bootstrap required, opened a new log
I20260926 20:10:31.319118 26010 ts_tablet_manager.cc:1397] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:10:31.319182 26012 log.cc:824] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:31.320389 26011 tablet_bootstrap.cc:492] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80: No bootstrap required, opened a new log
I20260926 20:10:31.320602 26011 ts_tablet_manager.cc:1397] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:10:31.320844 26012 tablet_bootstrap.cc:492] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5: No bootstrap required, opened a new log
I20260926 20:10:31.321027 26012 ts_tablet_manager.cc:1397] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 20:10:31.323817 26012 raft_consensus.cc:348] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } }
I20260926 20:10:31.324056 26012 raft_consensus.cc:374] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:31.323884 26010 raft_consensus.cc:348] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } }
I20260926 20:10:31.324107 26012 raft_consensus.cc:729] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a522790944e41e4bece9ca987e64bf5, State: Initialized, Role: FOLLOWER
I20260926 20:10:31.324126 26010 raft_consensus.cc:374] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:31.324184 26010 raft_consensus.cc:729] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2427ff8243d44f7099b5d1d94db18d36, State: Initialized, Role: FOLLOWER
I20260926 20:10:31.324536 26012 consensus_queue.cc:260] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } }
I20260926 20:10:31.324438 26011 raft_consensus.cc:348] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } }
I20260926 20:10:31.324671 26011 raft_consensus.cc:374] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:31.324589 26010 consensus_queue.cc:260] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } }
I20260926 20:10:31.324733 26011 raft_consensus.cc:729] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8648836dffb143cbb04148defddfed80, State: Initialized, Role: FOLLOWER
I20260926 20:10:31.325212 25987 heartbeater.cc:503] Master 127.23.50.190:33455 was elected leader, sending a full tablet report...
I20260926 20:10:31.325131 26011 consensus_queue.cc:260] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } }
I20260926 20:10:31.325578 26010 ts_tablet_manager.cc:1428] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:10:31.325928 25706 heartbeater.cc:503] Master 127.23.50.190:33455 was elected leader, sending a full tablet report...
I20260926 20:10:31.326030 26011 ts_tablet_manager.cc:1428] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260926 20:10:31.326323 25847 heartbeater.cc:503] Master 127.23.50.190:33455 was elected leader, sending a full tablet report...
I20260926 20:10:31.326925 26012 ts_tablet_manager.cc:1428] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:10:31.424069 26016 raft_consensus.cc:482] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:31.424304 26016 raft_consensus.cc:504] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } }
I20260926 20:10:31.425585 26016 leader_election.cc:288] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173), 8648836dffb143cbb04148defddfed80 (127.23.50.130:38489)
I20260926 20:10:31.429261 26005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.425913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:33193 (local address 127.23.50.129:37173)
0926 20:10:31.426074 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.426079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.426098 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:10:31.426234 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.426236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.426285 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.426377 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:31.426673 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.427177 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.428265 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.428681 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.428812 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.428869 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.429054 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.429128 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:10:31.429257 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.425782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:37173 (local address 127.23.50.131:33193)
0926 20:10:31.425996 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.426013 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.426107 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.426390 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.426393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.426437 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:31.426556 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.426561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.427298 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.427304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.428143 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.428152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.428806 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.428810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.428918 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.428947 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.429037 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.429114 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"spinlock_wait_cycles":8832}
I20260926 20:10:31.429992 25660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300b545cce99486aa8aa8d386186c0b4" candidate_uuid: "4a522790944e41e4bece9ca987e64bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2427ff8243d44f7099b5d1d94db18d36" is_pre_election: true
I20260926 20:10:31.430119 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.425895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:38489 (local address 127.23.50.131:51999)
0926 20:10:31.426279 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.426290 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.426461 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.426736 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.426739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.426754 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:31.427217 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.427225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.427943 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.427950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.428797 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.428815 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.429676 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.429679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.429786 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.429809 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.429913 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.429975 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"spinlock_wait_cycles":63104,"thread_start_us":141,"threads_started":1}
I20260926 20:10:31.430101 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.426002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:51999 (local address 127.23.50.130:38489)
0926 20:10:31.426114 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.426118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.426388 (+ 270us) server_negotiation.cc:407] Connection header received
0926 20:10:31.426575 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.426583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.426623 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.426723 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:31.427331 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.427832 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.428921 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.429546 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.429673 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.429737 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.429939 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.430005 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:10:31.430348 25660 raft_consensus.cc:2454] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a522790944e41e4bece9ca987e64bf5 in term 0.
I20260926 20:10:31.430874 25873 leader_election.cc:302] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2427ff8243d44f7099b5d1d94db18d36, 4a522790944e41e4bece9ca987e64bf5; no voters:
I20260926 20:10:31.430985 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300b545cce99486aa8aa8d386186c0b4" candidate_uuid: "4a522790944e41e4bece9ca987e64bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8648836dffb143cbb04148defddfed80" is_pre_election: true
I20260926 20:10:31.431160 26016 raft_consensus.cc:2781] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:31.431249 26016 raft_consensus.cc:482] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:31.431308 26016 raft_consensus.cc:3037] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:31.431361 25783 raft_consensus.cc:2454] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a522790944e41e4bece9ca987e64bf5 in term 0.
I20260926 20:10:31.432349 26016 raft_consensus.cc:504] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } }
I20260926 20:10:31.432715 26016 leader_election.cc:288] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [CANDIDATE]: Term 1 election: Requested vote from peers 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173), 8648836dffb143cbb04148defddfed80 (127.23.50.130:38489)
I20260926 20:10:31.433054 25660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300b545cce99486aa8aa8d386186c0b4" candidate_uuid: "4a522790944e41e4bece9ca987e64bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2427ff8243d44f7099b5d1d94db18d36"
I20260926 20:10:31.433082 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "300b545cce99486aa8aa8d386186c0b4" candidate_uuid: "4a522790944e41e4bece9ca987e64bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8648836dffb143cbb04148defddfed80"
I20260926 20:10:31.433166 25660 raft_consensus.cc:3037] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:31.433251 25783 raft_consensus.cc:3037] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:31.434300 25660 raft_consensus.cc:2454] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a522790944e41e4bece9ca987e64bf5 in term 1.
I20260926 20:10:31.434445 25783 raft_consensus.cc:2454] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a522790944e41e4bece9ca987e64bf5 in term 1.
I20260926 20:10:31.434672 25873 leader_election.cc:302] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2427ff8243d44f7099b5d1d94db18d36, 4a522790944e41e4bece9ca987e64bf5; no voters:
I20260926 20:10:31.434917 26016 raft_consensus.cc:2781] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:31.435168 26016 raft_consensus.cc:686] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 1 LEADER]: Becoming Leader. State: Replica: 4a522790944e41e4bece9ca987e64bf5, State: Running, Role: LEADER
I20260926 20:10:31.435499 26016 consensus_queue.cc:237] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } }
I20260926 20:10:31.438588 25501 catalog_manager.cc:5495] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a522790944e41e4bece9ca987e64bf5 (127.23.50.131). New cstate: current_term: 1 leader_uuid: "4a522790944e41e4bece9ca987e64bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } health_report { overall_health: UNKNOWN } } }
Sep 26 20:10:31 dist-test-slave-7034 krb5kdc[25456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453430, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:31.475855 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.467591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:33455 (local address 127.0.0.1:39784)
0926 20:10:31.467893 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.467905 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.468052 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.468345 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.468348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.468758 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:31.468845 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.468851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.469589 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.469593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.470203 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.470212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.470637 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.470640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.470721 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.471240 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:31.471251 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:31.472681 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:31.474286 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:31.474296 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:31.474302 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:31.474540 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:31.474813 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:31.474816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:31.474818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:31.474896 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:31.475271 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:31.475280 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:31.475394 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.475505 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.475662 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":115,"threads_started":1}
I20260926 20:10:31.475854 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.467629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39784 (local address 127.23.50.190:33455)
0926 20:10:31.467768 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.467775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.467983 (+ 208us) server_negotiation.cc:407] Connection header received
0926 20:10:31.468165 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.468168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.468213 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.468298 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:31.468961 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.469477 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.470315 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.470528 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.470635 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.472825 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:31.472829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:31.472831 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:31.472850 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:31.474175 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:31.474636 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:31.474642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:31.474644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:31.474720 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:31.474980 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:31.474986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:31.474988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:31.475158 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:31.475260 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.475562 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.475713 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":46}
I20260926 20:10:31.479946 25501 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:39784:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:10:31.480681 25501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:31.485774 25781 tablet_service.cc:1467] Processing CreateTablet for tablet 4e079309d30e4dcc8ac461ea8cba935f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d9d45208524e4dc88ed9780de4fc8774]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:10:31.485649 25640 tablet_service.cc:1467] Processing CreateTablet for tablet 4e079309d30e4dcc8ac461ea8cba935f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d9d45208524e4dc88ed9780de4fc8774]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:10:31.486443 25781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e079309d30e4dcc8ac461ea8cba935f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:31.486443 25640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e079309d30e4dcc8ac461ea8cba935f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:31.486073 25921 tablet_service.cc:1467] Processing CreateTablet for tablet 4e079309d30e4dcc8ac461ea8cba935f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d9d45208524e4dc88ed9780de4fc8774]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:10:31.486719 25921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e079309d30e4dcc8ac461ea8cba935f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:31.489583 26012 tablet_bootstrap.cc:492] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5: Bootstrap starting.
I20260926 20:10:31.489775 26011 tablet_bootstrap.cc:492] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: Bootstrap starting.
I20260926 20:10:31.490597 26012 tablet_bootstrap.cc:654] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:31.490712 26011 tablet_bootstrap.cc:654] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:31.491442 26010 tablet_bootstrap.cc:492] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36: Bootstrap starting.
I20260926 20:10:31.492343 26010 tablet_bootstrap.cc:654] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:31.492627 26012 tablet_bootstrap.cc:492] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5: No bootstrap required, opened a new log
I20260926 20:10:31.492724 26012 ts_tablet_manager.cc:1397] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:10:31.493247 26012 raft_consensus.cc:348] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.493382 26011 tablet_bootstrap.cc:492] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: No bootstrap required, opened a new log
I20260926 20:10:31.493391 26012 raft_consensus.cc:374] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:31.493430 26012 raft_consensus.cc:729] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a522790944e41e4bece9ca987e64bf5, State: Initialized, Role: FOLLOWER
I20260926 20:10:31.493475 26011 ts_tablet_manager.cc:1397] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:10:31.493568 26012 consensus_queue.cc:260] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.493904 26012 ts_tablet_manager.cc:1428] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260926 20:10:31.493831 26016 tablet_replica.cc:439] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } }
I20260926 20:10:31.493993 26016 tablet_replica.cc:442] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:10:31.493980 26011 raft_consensus.cc:348] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.494093 26011 raft_consensus.cc:374] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:31.494131 26011 raft_consensus.cc:729] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8648836dffb143cbb04148defddfed80, State: Initialized, Role: FOLLOWER
I20260926 20:10:31.494268 26011 consensus_queue.cc:260] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.494529 26018 tablet_replica.cc:439] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } }
W20260926 20:10:31.494634 25988 tablet.cc:2369] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:10:31.494679 26018 tablet_replica.cc:442] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:10:31.495677 26010 tablet_bootstrap.cc:492] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36: No bootstrap required, opened a new log
I20260926 20:10:31.495798 26010 ts_tablet_manager.cc:1397] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:10:31.496308 26010 raft_consensus.cc:348] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.496466 26010 raft_consensus.cc:374] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260926 20:10:31.496472 25848 tablet.cc:2369] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:10:31.496513 26010 raft_consensus.cc:729] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2427ff8243d44f7099b5d1d94db18d36, State: Initialized, Role: FOLLOWER
I20260926 20:10:31.496636 26010 consensus_queue.cc:260] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.496929 26010 ts_tablet_manager.cc:1428] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:10:31.494551 26011 ts_tablet_manager.cc:1428] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:10:31.496944 26017 tablet_replica.cc:439] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } }
I20260926 20:10:31.497216 26017 tablet_replica.cc:442] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 20:10:31.523473 25707 tablet.cc:2369] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:10:31.704697 26018 raft_consensus.cc:482] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:31.704856 26018 raft_consensus.cc:504] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.706144 26018 leader_election.cc:288] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173), 4a522790944e41e4bece9ca987e64bf5 (127.23.50.131:37943)
I20260926 20:10:31.710035 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.706449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:58657 (local address 127.23.50.131:37943)
0926 20:10:31.706571 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.706577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.706883 (+ 306us) server_negotiation.cc:407] Connection header received
0926 20:10:31.707003 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.707008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.707058 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.707134 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:31.707413 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.707906 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.709051 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.709430 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.709526 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.709603 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.709798 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.709909 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:10:31.710070 26005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.706468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:45409 (local address 127.23.50.129:37173)
0926 20:10:31.706597 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.706602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.706709 (+ 107us) server_negotiation.cc:407] Connection header received
0926 20:10:31.706881 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.706885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.706937 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.707015 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:31.707401 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.707903 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.709058 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.709466 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.709601 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.709653 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.709890 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.709962 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:10:31.710052 26031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.706395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:37943 (local address 127.23.50.130:58657)
0926 20:10:31.706763 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.706783 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.706902 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.707162 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.707164 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.707178 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:31.707286 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.707291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.708017 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.708023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.708927 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.708936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.709549 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.709555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.709650 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.709684 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.709761 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.709880 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":59,"thread_start_us":63,"threads_started":1}
I20260926 20:10:31.710067 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.706314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:37173 (local address 127.23.50.130:45409)
0926 20:10:31.706624 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.706638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.706755 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.707037 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.707040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.707058 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:31.707274 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.707282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.708022 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.708025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.708924 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.708933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.709609 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.709616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.709718 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.709735 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.709832 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.709933 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":62,"thread_start_us":85,"threads_started":1}
I20260926 20:10:31.710736 25660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e079309d30e4dcc8ac461ea8cba935f" candidate_uuid: "8648836dffb143cbb04148defddfed80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2427ff8243d44f7099b5d1d94db18d36" is_pre_election: true
I20260926 20:10:31.710868 25660 raft_consensus.cc:2454] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8648836dffb143cbb04148defddfed80 in term 0.
I20260926 20:10:31.710875 25938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e079309d30e4dcc8ac461ea8cba935f" candidate_uuid: "8648836dffb143cbb04148defddfed80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a522790944e41e4bece9ca987e64bf5" is_pre_election: true
I20260926 20:10:31.711143 25938 raft_consensus.cc:2454] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8648836dffb143cbb04148defddfed80 in term 0.
I20260926 20:10:31.711266 25732 leader_election.cc:302] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2427ff8243d44f7099b5d1d94db18d36, 8648836dffb143cbb04148defddfed80; no voters:
I20260926 20:10:31.711541 26018 raft_consensus.cc:2781] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:31.711637 26018 raft_consensus.cc:482] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:31.711689 26018 raft_consensus.cc:3037] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:31.712621 26018 raft_consensus.cc:504] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.713016 26018 leader_election.cc:288] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [CANDIDATE]: Term 1 election: Requested vote from peers 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173), 4a522790944e41e4bece9ca987e64bf5 (127.23.50.131:37943)
I20260926 20:10:31.713375 25938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e079309d30e4dcc8ac461ea8cba935f" candidate_uuid: "8648836dffb143cbb04148defddfed80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a522790944e41e4bece9ca987e64bf5"
I20260926 20:10:31.713449 25660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e079309d30e4dcc8ac461ea8cba935f" candidate_uuid: "8648836dffb143cbb04148defddfed80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2427ff8243d44f7099b5d1d94db18d36"
I20260926 20:10:31.713515 25938 raft_consensus.cc:3037] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:31.713586 25660 raft_consensus.cc:3037] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:31.714576 25938 raft_consensus.cc:2454] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8648836dffb143cbb04148defddfed80 in term 1.
I20260926 20:10:31.714577 25660 raft_consensus.cc:2454] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8648836dffb143cbb04148defddfed80 in term 1.
I20260926 20:10:31.714910 25734 leader_election.cc:302] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2427ff8243d44f7099b5d1d94db18d36, 4a522790944e41e4bece9ca987e64bf5, 8648836dffb143cbb04148defddfed80; no voters:
I20260926 20:10:31.715113 26018 raft_consensus.cc:2781] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:31.715339 26018 raft_consensus.cc:686] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [term 1 LEADER]: Becoming Leader. State: Replica: 8648836dffb143cbb04148defddfed80, State: Running, Role: LEADER
I20260926 20:10:31.715684 26018 consensus_queue.cc:237] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.717973 26033 tablet_replica.cc:439] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: TxnStatusTablet state changed. Reason: New leader 8648836dffb143cbb04148defddfed80. Latest consensus state: current_term: 1 leader_uuid: "8648836dffb143cbb04148defddfed80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } }
I20260926 20:10:31.718113 26033 tablet_replica.cc:442] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:10:31.718432 26035 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:10:31.718650 26035 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:10:31.719431 25501 catalog_manager.cc:5495] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8648836dffb143cbb04148defddfed80 (127.23.50.130). New cstate: current_term: 1 leader_uuid: "8648836dffb143cbb04148defddfed80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } health_report { overall_health: UNKNOWN } } }
I20260926 20:10:31.736428 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.732154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:38489 (local address 127.0.0.1:59142)
0926 20:10:31.732356 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.732374 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.732484 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.732797 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.732799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.732805 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:10:31.732915 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.732919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.733965 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.733967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.734571 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.734579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.735160 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.735163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.735232 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.735278 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:10:31.736097 (+ 819us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:10:31.736115 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.736199 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.736283 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 20:10:31.736429 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.732281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.23.50.130:38489)
0926 20:10:31.732422 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.732427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.732447 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:10:31.732607 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.732610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.732673 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.732777 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:31.733020 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.733807 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.734686 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.735038 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.735156 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.735398 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:31.735969 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:31.736084 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.736232 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.736292 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:10:31.742483 25660 raft_consensus.cc:1261] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Refusing update from remote peer 8648836dffb143cbb04148defddfed80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:31.742483 25938 raft_consensus.cc:1261] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Refusing update from remote peer 8648836dffb143cbb04148defddfed80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:31.743147 26033 consensus_queue.cc:1035] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:31.743321 26018 consensus_queue.cc:1035] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:31.746995 26016 tablet_replica.cc:439] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5: TxnStatusTablet state changed. Reason: New leader 8648836dffb143cbb04148defddfed80. Latest consensus state: current_term: 1 leader_uuid: "8648836dffb143cbb04148defddfed80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } }
I20260926 20:10:31.747135 26016 tablet_replica.cc:442] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:10:31.747314 26017 tablet_replica.cc:439] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36: TxnStatusTablet state changed. Reason: New leader 8648836dffb143cbb04148defddfed80. Latest consensus state: current_term: 1 leader_uuid: "8648836dffb143cbb04148defddfed80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } }
I20260926 20:10:31.747429 26017 tablet_replica.cc:442] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:10:31.751492 26016 tablet_replica.cc:439] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8648836dffb143cbb04148defddfed80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } }
I20260926 20:10:31.751616 26016 tablet_replica.cc:442] T 4e079309d30e4dcc8ac461ea8cba935f P 4a522790944e41e4bece9ca987e64bf5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:10:31.751631 26017 tablet_replica.cc:439] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8648836dffb143cbb04148defddfed80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } }
I20260926 20:10:31.751746 26017 tablet_replica.cc:442] T 4e079309d30e4dcc8ac461ea8cba935f P 2427ff8243d44f7099b5d1d94db18d36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:10:31.751931 26039 mvcc.cc:205] Tried to move back new op lower bound from 7333697256408997888 to 7333697256310190080. Current Snapshot: MvccSnapshot[applied={T|T < 7333697256408997888}]
I20260926 20:10:31.753669 26038 mvcc.cc:205] Tried to move back new op lower bound from 7333697256408997888 to 7333697256310190080. Current Snapshot: MvccSnapshot[applied={T|T < 7333697256408997888}]
I20260926 20:10:31.754014 26043 tablet_replica.cc:439] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8648836dffb143cbb04148defddfed80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } }
I20260926 20:10:31.754036 26033 tablet_replica.cc:439] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8648836dffb143cbb04148defddfed80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } } }
I20260926 20:10:31.754158 26043 tablet_replica.cc:442] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:10:31.754168 26033 tablet_replica.cc:442] T 4e079309d30e4dcc8ac461ea8cba935f P 8648836dffb143cbb04148defddfed80: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:10:31.765332 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.761619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33314 (local address 127.23.50.131:37943)
0926 20:10:31.761782 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.761787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.761802 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:10:31.761947 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.761950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.762003 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.762119 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:31.762375 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.762997 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.763852 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.764075 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.764186 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.764494 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:31.764939 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:31.765067 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.765129 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.765183 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 20:10:31.783582 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.775388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:43963 (local address 127.23.50.130:38489)
0926 20:10:31.775506 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.775510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.775927 (+ 417us) server_negotiation.cc:407] Connection header received
0926 20:10:31.776119 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.776122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.776192 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.776268 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:31.779722 (+ 3454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.780539 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.781989 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.782323 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.782405 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.783073 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:31.783162 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:31.783202 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.783371 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.783436 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:10:31.784313 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.774997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:38489 (local address 127.23.50.131:43963)
0926 20:10:31.775577 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.775595 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.776006 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.779422 (+ 3416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.779426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.779438 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:10:31.779576 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.779583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.780671 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.780675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.781802 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.781815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.782529 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.782532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.782631 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.782893 (+ 262us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:10:31.783268 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:10:31.783274 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.783820 (+ 546us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.783911 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":106,"threads_started":1}
I20260926 20:10:31.796041 25660 raft_consensus.cc:1261] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Refusing update from remote peer 4a522790944e41e4bece9ca987e64bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:31.796286 25783 raft_consensus.cc:1261] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Refusing update from remote peer 4a522790944e41e4bece9ca987e64bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:31.796569 26020 consensus_queue.cc:1035] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:31.796847 26016 consensus_queue.cc:1035] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:31.825439 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.819951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:37943 (local address 127.23.50.130:34385)
0926 20:10:31.820914 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.820930 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.821034 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.821312 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.821315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.821323 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:10:31.821457 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.821463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.822652 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.822655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.823895 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.823907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.824557 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.824561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.824657 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.824717 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:10:31.825117 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:10:31.825124 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.825190 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.825273 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":76,"threads_started":1}
I20260926 20:10:31.825459 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.820261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:34385 (local address 127.23.50.131:37943)
0926 20:10:31.820397 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.820401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.821020 (+ 619us) server_negotiation.cc:407] Connection header received
0926 20:10:31.821124 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.821129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.821197 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.821290 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:31.821570 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.822515 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.824033 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.824367 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.824441 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.824904 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:31.825012 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:31.825104 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.825218 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.825290 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:10:31.852065 25500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:31.852274 25500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:31.855235 25500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070: Sending DeleteTablet for 3 replicas of tablet 300b545cce99486aa8aa8d386186c0b4
I20260926 20:10:31.856043 25640 tablet_service.cc:1514] Processing DeleteTablet for tablet 300b545cce99486aa8aa8d386186c0b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:31 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:34968
I20260926 20:10:31.856189 25781 tablet_service.cc:1514] Processing DeleteTablet for tablet 300b545cce99486aa8aa8d386186c0b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:31 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:59140
I20260926 20:10:31.856664 26057 tablet_replica.cc:330] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36: stopping tablet replica
I20260926 20:10:31.856909 26058 tablet_replica.cc:330] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80: stopping tablet replica
I20260926 20:10:31.857161 25921 tablet_service.cc:1514] Processing DeleteTablet for tablet 300b545cce99486aa8aa8d386186c0b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:31 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:33300
I20260926 20:10:31.857561 26058 raft_consensus.cc:2229] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:31.857698 26057 raft_consensus.cc:2229] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:31.858049 26057 raft_consensus.cc:2258] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:31.858237 26058 raft_consensus.cc:2258] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:31.858266 25500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:10:31.858719 25500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:31.860229 26058 ts_tablet_manager.cc:1905] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:31.861224 26057 ts_tablet_manager.cc:1905] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:31.864125 25640 tablet_service.cc:1467] Processing CreateTablet for tablet e0a4d12112a04daaaffb8168569918cf (DEFAULT_TABLE table=test-table [id=9d0e18e9dcd74cea8a5a21e9e60f8e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:31.864408 25640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a4d12112a04daaaffb8168569918cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:31.866882 26010 tablet_bootstrap.cc:492] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36: Bootstrap starting.
I20260926 20:10:31.867559 26058 ts_tablet_manager.cc:1918] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:10:31.867844 26010 tablet_bootstrap.cc:654] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:31.867877 26057 ts_tablet_manager.cc:1918] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:10:31.868093 26057 log.cc:1196] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/wal/wals/300b545cce99486aa8aa8d386186c0b4
I20260926 20:10:31.868513 26057 ts_tablet_manager.cc:1939] T 300b545cce99486aa8aa8d386186c0b4 P 2427ff8243d44f7099b5d1d94db18d36: Deleting consensus metadata
I20260926 20:10:31.868643 26058 log.cc:1196] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/300b545cce99486aa8aa8d386186c0b4
I20260926 20:10:31.869037 26058 ts_tablet_manager.cc:1939] T 300b545cce99486aa8aa8d386186c0b4 P 8648836dffb143cbb04148defddfed80: Deleting consensus metadata
I20260926 20:10:31.870314 25921 tablet_service.cc:1467] Processing CreateTablet for tablet e0a4d12112a04daaaffb8168569918cf (DEFAULT_TABLE table=test-table [id=9d0e18e9dcd74cea8a5a21e9e60f8e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:31.870553 25487 catalog_manager.cc:4841] TS 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173): tablet 300b545cce99486aa8aa8d386186c0b4 (table test-table [id=27995b6691484b7e96f0c0789fdc5af6]) successfully deleted
I20260926 20:10:31.870641 25921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a4d12112a04daaaffb8168569918cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:31.869405 25781 tablet_service.cc:1467] Processing CreateTablet for tablet e0a4d12112a04daaaffb8168569918cf (DEFAULT_TABLE table=test-table [id=9d0e18e9dcd74cea8a5a21e9e60f8e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:31.871011 25781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a4d12112a04daaaffb8168569918cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:31.873311 26012 tablet_bootstrap.cc:492] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5: Bootstrap starting.
I20260926 20:10:31.873342 26011 tablet_bootstrap.cc:492] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80: Bootstrap starting.
I20260926 20:10:31.874346 26012 tablet_bootstrap.cc:654] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:31.874701 26011 tablet_bootstrap.cc:654] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:31.875720 26010 tablet_bootstrap.cc:492] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36: No bootstrap required, opened a new log
I20260926 20:10:31.875844 26010 ts_tablet_manager.cc:1397] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260926 20:10:31.876369 26010 raft_consensus.cc:348] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.876528 26010 raft_consensus.cc:374] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:31.876569 26010 raft_consensus.cc:729] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2427ff8243d44f7099b5d1d94db18d36, State: Initialized, Role: FOLLOWER
I20260926 20:10:31.876695 26010 consensus_queue.cc:260] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.876984 26010 ts_tablet_manager.cc:1428] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 20:10:31.877362 25488 catalog_manager.cc:4841] TS 8648836dffb143cbb04148defddfed80 (127.23.50.130:38489): tablet 300b545cce99486aa8aa8d386186c0b4 (table test-table [id=27995b6691484b7e96f0c0789fdc5af6]) successfully deleted
I20260926 20:10:31.877022 26059 tablet_replica.cc:330] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5: stopping tablet replica
I20260926 20:10:31.878103 26059 raft_consensus.cc:2229] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:10:31.878505 26059 raft_consensus.cc:2258] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:31.879868 26059 ts_tablet_manager.cc:1905] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:31.880650 26011 tablet_bootstrap.cc:492] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80: No bootstrap required, opened a new log
I20260926 20:10:31.880733 26011 ts_tablet_manager.cc:1397] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 20:10:31.881140 26011 raft_consensus.cc:348] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.881246 26011 raft_consensus.cc:374] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:31.881276 26011 raft_consensus.cc:729] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8648836dffb143cbb04148defddfed80, State: Initialized, Role: FOLLOWER
I20260926 20:10:31.881393 26011 consensus_queue.cc:260] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.881646 26011 ts_tablet_manager.cc:1428] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 20:10:31.884799 26059 ts_tablet_manager.cc:1918] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:10:31.884945 26059 log.cc:1196] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/300b545cce99486aa8aa8d386186c0b4
I20260926 20:10:31.885324 26059 ts_tablet_manager.cc:1939] T 300b545cce99486aa8aa8d386186c0b4 P 4a522790944e41e4bece9ca987e64bf5: Deleting consensus metadata
I20260926 20:10:31.886584 25489 catalog_manager.cc:4841] TS 4a522790944e41e4bece9ca987e64bf5 (127.23.50.131:37943): tablet 300b545cce99486aa8aa8d386186c0b4 (table test-table [id=27995b6691484b7e96f0c0789fdc5af6]) successfully deleted
I20260926 20:10:31.886642 26012 tablet_bootstrap.cc:492] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5: No bootstrap required, opened a new log
I20260926 20:10:31.886718 26012 ts_tablet_manager.cc:1397] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20260926 20:10:31.887215 26012 raft_consensus.cc:348] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.887344 26012 raft_consensus.cc:374] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:31.887382 26012 raft_consensus.cc:729] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a522790944e41e4bece9ca987e64bf5, State: Initialized, Role: FOLLOWER
I20260926 20:10:31.887514 26012 consensus_queue.cc:260] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.887796 26012 ts_tablet_manager.cc:1428] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:10:31.944837 26018 raft_consensus.cc:482] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:31.944999 26018 raft_consensus.cc:504] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.945482 26018 leader_election.cc:288] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173), 4a522790944e41e4bece9ca987e64bf5 (127.23.50.131:37943)
I20260926 20:10:31.946296 25660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a4d12112a04daaaffb8168569918cf" candidate_uuid: "8648836dffb143cbb04148defddfed80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2427ff8243d44f7099b5d1d94db18d36" is_pre_election: true
I20260926 20:10:31.946496 25660 raft_consensus.cc:2454] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8648836dffb143cbb04148defddfed80 in term 0.
I20260926 20:10:31.946401 25938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a4d12112a04daaaffb8168569918cf" candidate_uuid: "8648836dffb143cbb04148defddfed80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a522790944e41e4bece9ca987e64bf5" is_pre_election: true
I20260926 20:10:31.946643 25938 raft_consensus.cc:2454] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8648836dffb143cbb04148defddfed80 in term 0.
I20260926 20:10:31.946903 25732 leader_election.cc:302] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2427ff8243d44f7099b5d1d94db18d36, 8648836dffb143cbb04148defddfed80; no voters:
I20260926 20:10:31.947110 26018 raft_consensus.cc:2781] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:31.947184 26018 raft_consensus.cc:482] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:31.947234 26018 raft_consensus.cc:3037] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:31.948503 26018 raft_consensus.cc:504] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.948897 26018 leader_election.cc:288] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [CANDIDATE]: Term 1 election: Requested vote from peers 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173), 4a522790944e41e4bece9ca987e64bf5 (127.23.50.131:37943)
I20260926 20:10:31.949252 25938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a4d12112a04daaaffb8168569918cf" candidate_uuid: "8648836dffb143cbb04148defddfed80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a522790944e41e4bece9ca987e64bf5"
I20260926 20:10:31.949389 25938 raft_consensus.cc:3037] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:31.949597 25660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a4d12112a04daaaffb8168569918cf" candidate_uuid: "8648836dffb143cbb04148defddfed80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2427ff8243d44f7099b5d1d94db18d36"
I20260926 20:10:31.949729 25660 raft_consensus.cc:3037] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:31.950598 25938 raft_consensus.cc:2454] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8648836dffb143cbb04148defddfed80 in term 1.
I20260926 20:10:31.950910 25734 leader_election.cc:302] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a522790944e41e4bece9ca987e64bf5, 8648836dffb143cbb04148defddfed80; no voters:
I20260926 20:10:31.950959 25660 raft_consensus.cc:2454] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8648836dffb143cbb04148defddfed80 in term 1.
I20260926 20:10:31.951176 26018 raft_consensus.cc:2781] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:31.951279 26018 raft_consensus.cc:686] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 1 LEADER]: Becoming Leader. State: Replica: 8648836dffb143cbb04148defddfed80, State: Running, Role: LEADER
I20260926 20:10:31.951414 26018 consensus_queue.cc:237] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:31.953460 25501 catalog_manager.cc:5495] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8648836dffb143cbb04148defddfed80 (127.23.50.130). New cstate: current_term: 1 leader_uuid: "8648836dffb143cbb04148defddfed80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } health_report { overall_health: UNKNOWN } } }
I20260926 20:10:31.977648 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.973910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.23.50.130:38489)
0926 20:10:31.974033 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.974037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.974058 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:10:31.974214 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.974217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.974279 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.974369 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:31.974627 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.975367 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.976312 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.976551 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.976669 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.976925 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:31.977040 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:31.977184 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.977413 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.977479 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:10:31.987233 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.983812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:42501 (local address 127.23.50.130:38489)
0926 20:10:31.983921 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:31.983924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:31.983956 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:10:31.984106 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:31.984108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:31.984148 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:31.984255 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:31.984500 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.985017 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.986203 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.986412 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.986498 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.986787 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:31.986851 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:31.986907 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:31.987040 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 20:10:31.987082 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:10:31.987257 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:31.983742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:38489 (local address 127.23.50.130:42501)
0926 20:10:31.983892 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:31.983909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:31.983999 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:31.984270 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:31.984274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:31.984282 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:10:31.984384 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.984390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.985137 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.985140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.986054 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:31.986064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:31.986536 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:31.986540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:31.986626 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:31.986659 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:10:31.986942 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:10:31.986949 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:10:31.987025 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:10:31.987110 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 20:10:31.993280 25938 raft_consensus.cc:1261] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Refusing update from remote peer 8648836dffb143cbb04148defddfed80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:31.993412 25660 raft_consensus.cc:1261] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Refusing update from remote peer 8648836dffb143cbb04148defddfed80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:31.993719 26018 consensus_queue.cc:1035] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:31.993897 26043 consensus_queue.cc:1035] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:31.999583 26040 mvcc.cc:205] Tried to move back new op lower bound from 7333697257440399360 to 7333697257274318848. Current Snapshot: MvccSnapshot[applied={T|T < 7333697257440399360}]
W20260926 20:10:32.015901 26038 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:10:33.576836 25500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:33.576994 25500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:33.579813 25500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070: Sending DeleteTablet for 3 replicas of tablet e0a4d12112a04daaaffb8168569918cf
I20260926 20:10:33.580466 25640 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a4d12112a04daaaffb8168569918cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:33 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:34968
I20260926 20:10:33.580528 25780 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a4d12112a04daaaffb8168569918cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:33 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:59140
I20260926 20:10:33.580583 25921 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a4d12112a04daaaffb8168569918cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:33 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:33300
I20260926 20:10:33.580847 26087 tablet_replica.cc:330] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36: stopping tablet replica
I20260926 20:10:33.580996 26087 raft_consensus.cc:2229] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:33.581123 26089 tablet_replica.cc:330] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5: stopping tablet replica
I20260926 20:10:33.581146 26087 raft_consensus.cc:2258] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:33.581265 26089 raft_consensus.cc:2229] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:33.581390 26089 raft_consensus.cc:2258] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:33.582091 26089 ts_tablet_manager.cc:1905] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:33.584005 26087 ts_tablet_manager.cc:1905] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:33.584223 26088 tablet_replica.cc:330] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80: stopping tablet replica
I20260926 20:10:33.584416 26088 raft_consensus.cc:2229] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:10:33.584793 26088 raft_consensus.cc:2258] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:33.585498 26088 ts_tablet_manager.cc:1905] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:33.585510 26089 ts_tablet_manager.cc:1918] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:10:33.585578 26089 log.cc:1196] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/e0a4d12112a04daaaffb8168569918cf
I20260926 20:10:33.585894 26089 ts_tablet_manager.cc:1939] T e0a4d12112a04daaaffb8168569918cf P 4a522790944e41e4bece9ca987e64bf5: Deleting consensus metadata
I20260926 20:10:33.586782 26087 ts_tablet_manager.cc:1918] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:10:33.586869 26087 log.cc:1196] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/wal/wals/e0a4d12112a04daaaffb8168569918cf
I20260926 20:10:33.587025 25489 catalog_manager.cc:4841] TS 4a522790944e41e4bece9ca987e64bf5 (127.23.50.131:37943): tablet e0a4d12112a04daaaffb8168569918cf (table test-table [id=9d0e18e9dcd74cea8a5a21e9e60f8e16]) successfully deleted
I20260926 20:10:33.587181 26087 ts_tablet_manager.cc:1939] T e0a4d12112a04daaaffb8168569918cf P 2427ff8243d44f7099b5d1d94db18d36: Deleting consensus metadata
I20260926 20:10:33.587908 25487 catalog_manager.cc:4841] TS 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173): tablet e0a4d12112a04daaaffb8168569918cf (table test-table [id=9d0e18e9dcd74cea8a5a21e9e60f8e16]) successfully deleted
I20260926 20:10:33.588490 26088 ts_tablet_manager.cc:1918] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:10:33.588586 26088 log.cc:1196] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/e0a4d12112a04daaaffb8168569918cf
I20260926 20:10:33.588886 26088 ts_tablet_manager.cc:1939] T e0a4d12112a04daaaffb8168569918cf P 8648836dffb143cbb04148defddfed80: Deleting consensus metadata
I20260926 20:10:33.589743 25488 catalog_manager.cc:4841] TS 8648836dffb143cbb04148defddfed80 (127.23.50.130:38489): tablet e0a4d12112a04daaaffb8168569918cf (table test-table [id=9d0e18e9dcd74cea8a5a21e9e60f8e16]) successfully deleted
Sep 26 20:10:33 dist-test-slave-7034 krb5kdc[25456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453433, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:10:33 dist-test-slave-7034 krb5kdc[25456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453433, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:33.618595 26097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:33.599825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.23.50.190:33455)
0926 20:10:33.600160 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:33.600167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:33.600183 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:10:33.600295 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:33.600298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:33.600342 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:33.600423 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:33.601160 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.601721 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.602710 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.602941 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.603093 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:33.614777 (+ 11684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:33.614789 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:33.614791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:33.614827 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:33.616467 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:33.616958 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:33.616961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:33.616962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:33.617064 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:33.617491 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:33.617494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:33.617496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:33.617738 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:33.617920 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:33.618226 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 20:10:33.618351 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":130,"threads_started":1}
I20260926 20:10:33.621518 25500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:10:33.621999 25500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:33.626930 25640 tablet_service.cc:1467] Processing CreateTablet for tablet 52298dbe43454f079b701d1e5a67a446 (DEFAULT_TABLE table=test-table [id=1cfe182af51c45e1a7d498761ee61957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:33.627049 25780 tablet_service.cc:1467] Processing CreateTablet for tablet 52298dbe43454f079b701d1e5a67a446 (DEFAULT_TABLE table=test-table [id=1cfe182af51c45e1a7d498761ee61957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:33.627102 25921 tablet_service.cc:1467] Processing CreateTablet for tablet 52298dbe43454f079b701d1e5a67a446 (DEFAULT_TABLE table=test-table [id=1cfe182af51c45e1a7d498761ee61957]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:33.627241 25640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52298dbe43454f079b701d1e5a67a446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:33.627377 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52298dbe43454f079b701d1e5a67a446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:33.627410 25921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52298dbe43454f079b701d1e5a67a446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:33.629580 26098 tablet_bootstrap.cc:492] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36: Bootstrap starting.
I20260926 20:10:33.630294 26099 tablet_bootstrap.cc:492] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5: Bootstrap starting.
I20260926 20:10:33.630321 26100 tablet_bootstrap.cc:492] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80: Bootstrap starting.
I20260926 20:10:33.630726 26098 tablet_bootstrap.cc:654] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:33.631098 26100 tablet_bootstrap.cc:654] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:33.631098 26099 tablet_bootstrap.cc:654] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:33.631943 26098 tablet_bootstrap.cc:492] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36: No bootstrap required, opened a new log
I20260926 20:10:33.632045 26098 ts_tablet_manager.cc:1397] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:10:33.632244 26099 tablet_bootstrap.cc:492] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5: No bootstrap required, opened a new log
I20260926 20:10:33.632360 26099 ts_tablet_manager.cc:1397] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:10:33.632798 26099 raft_consensus.cc:348] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.632923 26099 raft_consensus.cc:374] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:33.632969 26099 raft_consensus.cc:729] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a522790944e41e4bece9ca987e64bf5, State: Initialized, Role: FOLLOWER
I20260926 20:10:33.632941 26098 raft_consensus.cc:348] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.633069 26098 raft_consensus.cc:374] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:33.633116 26098 raft_consensus.cc:729] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2427ff8243d44f7099b5d1d94db18d36, State: Initialized, Role: FOLLOWER
I20260926 20:10:33.633127 26099 consensus_queue.cc:260] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.633312 26100 tablet_bootstrap.cc:492] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80: No bootstrap required, opened a new log
I20260926 20:10:33.633283 26098 consensus_queue.cc:260] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.633405 26100 ts_tablet_manager.cc:1397] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:10:33.633525 26099 ts_tablet_manager.cc:1428] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 20:10:33.633685 26098 ts_tablet_manager.cc:1428] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:33.633867 26100 raft_consensus.cc:348] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.633981 26100 raft_consensus.cc:374] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:33.634021 26100 raft_consensus.cc:729] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8648836dffb143cbb04148defddfed80, State: Initialized, Role: FOLLOWER
I20260926 20:10:33.634135 26100 consensus_queue.cc:260] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.634488 26100 ts_tablet_manager.cc:1428] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:10:33.659328 26105 raft_consensus.cc:482] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:33.659492 26105 raft_consensus.cc:504] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.660784 26105 leader_election.cc:288] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8648836dffb143cbb04148defddfed80 (127.23.50.130:38489), 4a522790944e41e4bece9ca987e64bf5 (127.23.50.131:37943)
I20260926 20:10:33.665270 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:33.661284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:34331 (local address 127.23.50.131:37943)
0926 20:10:33.661542 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:33.661555 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:33.661890 (+ 335us) server_negotiation.cc:407] Connection header received
0926 20:10:33.662056 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:33.662059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:33.662108 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:33.662207 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:33.662564 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.663065 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.664212 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.664636 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.664756 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:33.664812 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:33.665047 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 20:10:33.665121 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20260926 20:10:33.665283 26109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:33.661099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:37943 (local address 127.23.50.129:34331)
0926 20:10:33.661779 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:33.661799 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:33.661943 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:33.662316 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:33.662319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:33.662335 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:33.662446 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:33.662452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.663208 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.663213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:33.664080 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:33.664089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.664756 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.664759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:33.664866 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:33.664944 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 20:10:33.665002 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 20:10:33.665092 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":401,"thread_start_us":79,"threads_started":1}
I20260926 20:10:33.665280 26107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:33.661243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:57373 (local address 127.23.50.130:38489)
0926 20:10:33.661565 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:33.661574 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:33.661587 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:10:33.661745 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:33.661748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:33.661805 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:33.661932 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:33.662223 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.663028 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.664240 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.664656 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.664789 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:33.664845 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:33.665088 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 20:10:33.665165 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":133,"threads_started":1}
I20260926 20:10:33.665501 26106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:33.661102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:38489 (local address 127.23.50.129:57373)
0926 20:10:33.661464 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:33.661482 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:33.661590 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:33.661946 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:33.661949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:33.661967 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:33.662095 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:33.662104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.663141 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.663147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:33.664094 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:33.664103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.664800 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.664803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:33.664946 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:33.664963 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 20:10:33.665057 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:10:33.665121 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":54,"thread_start_us":141,"threads_started":1}
I20260926 20:10:33.665781 25938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52298dbe43454f079b701d1e5a67a446" candidate_uuid: "2427ff8243d44f7099b5d1d94db18d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a522790944e41e4bece9ca987e64bf5" is_pre_election: true
I20260926 20:10:33.665987 25938 raft_consensus.cc:2454] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2427ff8243d44f7099b5d1d94db18d36 in term 0.
I20260926 20:10:33.666064 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52298dbe43454f079b701d1e5a67a446" candidate_uuid: "2427ff8243d44f7099b5d1d94db18d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8648836dffb143cbb04148defddfed80" is_pre_election: true
I20260926 20:10:33.666221 25783 raft_consensus.cc:2454] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2427ff8243d44f7099b5d1d94db18d36 in term 0.
I20260926 20:10:33.666411 25593 leader_election.cc:302] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2427ff8243d44f7099b5d1d94db18d36, 4a522790944e41e4bece9ca987e64bf5; no voters:
I20260926 20:10:33.666689 26105 raft_consensus.cc:2781] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:33.666749 26105 raft_consensus.cc:482] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:33.666788 26105 raft_consensus.cc:3037] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:33.667775 26105 raft_consensus.cc:504] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.668087 26105 leader_election.cc:288] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [CANDIDATE]: Term 1 election: Requested vote from peers 8648836dffb143cbb04148defddfed80 (127.23.50.130:38489), 4a522790944e41e4bece9ca987e64bf5 (127.23.50.131:37943)
I20260926 20:10:33.668442 25938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52298dbe43454f079b701d1e5a67a446" candidate_uuid: "2427ff8243d44f7099b5d1d94db18d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a522790944e41e4bece9ca987e64bf5"
I20260926 20:10:33.668460 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52298dbe43454f079b701d1e5a67a446" candidate_uuid: "2427ff8243d44f7099b5d1d94db18d36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8648836dffb143cbb04148defddfed80"
I20260926 20:10:33.668591 25938 raft_consensus.cc:3037] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:33.668609 25783 raft_consensus.cc:3037] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:33.669618 25938 raft_consensus.cc:2454] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2427ff8243d44f7099b5d1d94db18d36 in term 1.
I20260926 20:10:33.669780 25783 raft_consensus.cc:2454] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2427ff8243d44f7099b5d1d94db18d36 in term 1.
I20260926 20:10:33.669929 25593 leader_election.cc:302] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2427ff8243d44f7099b5d1d94db18d36, 4a522790944e41e4bece9ca987e64bf5; no voters:
I20260926 20:10:33.670105 26105 raft_consensus.cc:2781] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:33.670315 26105 raft_consensus.cc:686] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 1 LEADER]: Becoming Leader. State: Replica: 2427ff8243d44f7099b5d1d94db18d36, State: Running, Role: LEADER
I20260926 20:10:33.670573 26105 consensus_queue.cc:237] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.672930 25502 catalog_manager.cc:5495] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129). New cstate: current_term: 1 leader_uuid: "2427ff8243d44f7099b5d1d94db18d36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } health_report { overall_health: UNKNOWN } } }
I20260926 20:10:33.692773 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:33.689136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.23.50.129:37173)
0926 20:10:33.689405 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:33.689412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:33.689424 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:10:33.689494 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:33.689496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:33.689547 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:33.689650 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:33.689921 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.690498 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.691376 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.691613 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.691694 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:33.691971 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:33.692436 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:33.692521 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:33.692591 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:10:33.692638 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":126,"threads_started":1}
I20260926 20:10:33.703969 26107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:33.700189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:55425 (local address 127.23.50.130:38489)
0926 20:10:33.700351 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:33.700354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:33.700452 (+ 98us) server_negotiation.cc:407] Connection header received
0926 20:10:33.700508 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:33.700511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:33.700563 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:33.700634 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:33.700998 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.701608 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.702673 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.702947 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.703033 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:33.703423 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:33.703520 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:33.703629 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:33.703766 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 20:10:33.703823 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 20:10:33.704025 26116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:33.699989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:38489 (local address 127.23.50.129:55425)
0926 20:10:33.700333 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:33.700353 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:33.700442 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:33.700689 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:33.700692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:33.700702 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:10:33.700825 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:33.700830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.701733 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.701738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:33.702556 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:33.702566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.703073 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.703076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:33.703198 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:33.703246 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:10:33.703634 (+ 388us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:10:33.703640 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:10:33.703728 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 20:10:33.703816 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":200,"threads_started":1}
I20260926 20:10:33.710793 25783 raft_consensus.cc:1261] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Refusing update from remote peer 2427ff8243d44f7099b5d1d94db18d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:33.710772 25938 raft_consensus.cc:1261] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Refusing update from remote peer 2427ff8243d44f7099b5d1d94db18d36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:33.711284 26105 consensus_queue.cc:1035] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:33.711462 26110 consensus_queue.cc:1035] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:33.730041 26113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:33.726312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:42115 (local address 127.23.50.129:37173)
0926 20:10:33.726441 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:33.726446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:33.726629 (+ 183us) server_negotiation.cc:407] Connection header received
0926 20:10:33.726776 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:33.726779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:33.726832 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:33.726910 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:33.727187 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.727721 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.728734 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.729099 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.729220 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:33.729545 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:33.729635 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:33.729724 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:33.729813 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 20:10:33.729894 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:10:33.730052 26120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:33.726255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:37173 (local address 127.23.50.130:42115)
0926 20:10:33.726544 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:33.726556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:33.726674 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:33.726932 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:33.726935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:33.726943 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:10:33.727081 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:33.727087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.727834 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.727848 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:33.728615 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:33.728626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.729206 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.729209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:33.729294 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:33.729395 (+ 101us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:10:33.729730 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:10:33.729739 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:10:33.729802 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 20:10:33.729922 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":120,"threads_started":1}
I20260926 20:10:33.754303 25502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:33.754496 25502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:33.756750 25502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070: Sending DeleteTablet for 3 replicas of tablet 52298dbe43454f079b701d1e5a67a446
I20260926 20:10:33.757382 25640 tablet_service.cc:1514] Processing DeleteTablet for tablet 52298dbe43454f079b701d1e5a67a446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:33 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:34968
I20260926 20:10:33.757427 25780 tablet_service.cc:1514] Processing DeleteTablet for tablet 52298dbe43454f079b701d1e5a67a446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:33 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:59140
I20260926 20:10:33.757522 25921 tablet_service.cc:1514] Processing DeleteTablet for tablet 52298dbe43454f079b701d1e5a67a446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:33 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:33300
I20260926 20:10:33.757577 26087 tablet_replica.cc:330] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36: stopping tablet replica
I20260926 20:10:33.757594 26088 tablet_replica.cc:330] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80: stopping tablet replica
I20260926 20:10:33.757702 26088 raft_consensus.cc:2229] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:33.757701 26087 raft_consensus.cc:2229] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:10:33.757725 26089 tablet_replica.cc:330] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5: stopping tablet replica
I20260926 20:10:33.757815 26088 raft_consensus.cc:2258] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:33.757898 26089 raft_consensus.cc:2229] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:33.758023 26089 raft_consensus.cc:2258] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:33.758121 26087 raft_consensus.cc:2258] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:33.758854 26087 ts_tablet_manager.cc:1905] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:33.758855 26088 ts_tablet_manager.cc:1905] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:33.761531 26088 ts_tablet_manager.cc:1918] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:10:33.761519 26087 ts_tablet_manager.cc:1918] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:10:33.761605 26088 log.cc:1196] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/52298dbe43454f079b701d1e5a67a446
I20260926 20:10:33.761583 26089 ts_tablet_manager.cc:1905] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:33.761605 26087 log.cc:1196] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-0/wal/wals/52298dbe43454f079b701d1e5a67a446
I20260926 20:10:33.761907 26088 ts_tablet_manager.cc:1939] T 52298dbe43454f079b701d1e5a67a446 P 8648836dffb143cbb04148defddfed80: Deleting consensus metadata
I20260926 20:10:33.762528 26087 ts_tablet_manager.cc:1939] T 52298dbe43454f079b701d1e5a67a446 P 2427ff8243d44f7099b5d1d94db18d36: Deleting consensus metadata
I20260926 20:10:33.762676 25488 catalog_manager.cc:4841] TS 8648836dffb143cbb04148defddfed80 (127.23.50.130:38489): tablet 52298dbe43454f079b701d1e5a67a446 (table test-table [id=1cfe182af51c45e1a7d498761ee61957]) successfully deleted
I20260926 20:10:33.764163 26089 ts_tablet_manager.cc:1918] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:10:33.764230 26089 log.cc:1196] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/52298dbe43454f079b701d1e5a67a446
I20260926 20:10:33.764477 26089 ts_tablet_manager.cc:1939] T 52298dbe43454f079b701d1e5a67a446 P 4a522790944e41e4bece9ca987e64bf5: Deleting consensus metadata
I20260926 20:10:33.764832 25487 catalog_manager.cc:4841] TS 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173): tablet 52298dbe43454f079b701d1e5a67a446 (table test-table [id=1cfe182af51c45e1a7d498761ee61957]) successfully deleted
I20260926 20:10:33.765372 25489 catalog_manager.cc:4841] TS 4a522790944e41e4bece9ca987e64bf5 (127.23.50.131:37943): tablet 52298dbe43454f079b701d1e5a67a446 (table test-table [id=1cfe182af51c45e1a7d498761ee61957]) successfully deleted
I20260926 20:10:33.765348 25502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:10:33.765949 25502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:33.771082 25640 tablet_service.cc:1467] Processing CreateTablet for tablet 2eae7896b21e4ffaae179e90e451137c (DEFAULT_TABLE table=test-table [id=4d66b23b09d84cd8b045913a6daffc11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:33.771252 25780 tablet_service.cc:1467] Processing CreateTablet for tablet 2eae7896b21e4ffaae179e90e451137c (DEFAULT_TABLE table=test-table [id=4d66b23b09d84cd8b045913a6daffc11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:33.771384 25640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eae7896b21e4ffaae179e90e451137c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:33.771554 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eae7896b21e4ffaae179e90e451137c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:33.773701 26098 tablet_bootstrap.cc:492] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36: Bootstrap starting.
I20260926 20:10:33.773747 26100 tablet_bootstrap.cc:492] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80: Bootstrap starting.
I20260926 20:10:33.774605 26100 tablet_bootstrap.cc:654] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:33.774772 26098 tablet_bootstrap.cc:654] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:33.775444 25921 tablet_service.cc:1467] Processing CreateTablet for tablet 2eae7896b21e4ffaae179e90e451137c (DEFAULT_TABLE table=test-table [id=4d66b23b09d84cd8b045913a6daffc11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:33.775709 25921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2eae7896b21e4ffaae179e90e451137c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:33.776166 26098 tablet_bootstrap.cc:492] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36: No bootstrap required, opened a new log
I20260926 20:10:33.776252 26098 ts_tablet_manager.cc:1397] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:10:33.776717 26098 raft_consensus.cc:348] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.776825 26098 raft_consensus.cc:374] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:33.776856 26098 raft_consensus.cc:729] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2427ff8243d44f7099b5d1d94db18d36, State: Initialized, Role: FOLLOWER
I20260926 20:10:33.776991 26098 consensus_queue.cc:260] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.777257 26098 ts_tablet_manager.cc:1428] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:10:33.778038 26099 tablet_bootstrap.cc:492] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5: Bootstrap starting.
I20260926 20:10:33.778861 26099 tablet_bootstrap.cc:654] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:33.780161 26099 tablet_bootstrap.cc:492] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5: No bootstrap required, opened a new log
I20260926 20:10:33.780249 26099 ts_tablet_manager.cc:1397] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:10:33.780617 26099 raft_consensus.cc:348] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.780731 26099 raft_consensus.cc:374] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:33.780781 26099 raft_consensus.cc:729] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a522790944e41e4bece9ca987e64bf5, State: Initialized, Role: FOLLOWER
I20260926 20:10:33.780910 26099 consensus_queue.cc:260] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.781173 26099 ts_tablet_manager.cc:1428] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:10:33.789372 26100 tablet_bootstrap.cc:492] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80: No bootstrap required, opened a new log
I20260926 20:10:33.789490 26100 ts_tablet_manager.cc:1397] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20260926 20:10:33.789999 26100 raft_consensus.cc:348] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.790135 26100 raft_consensus.cc:374] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:33.790167 26100 raft_consensus.cc:729] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8648836dffb143cbb04148defddfed80, State: Initialized, Role: FOLLOWER
I20260926 20:10:33.790299 26100 consensus_queue.cc:260] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.790561 26100 ts_tablet_manager.cc:1428] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:10:33.867899 26104 raft_consensus.cc:482] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:33.868088 26104 raft_consensus.cc:504] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.868597 26104 leader_election.cc:288] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173), 8648836dffb143cbb04148defddfed80 (127.23.50.130:38489)
I20260926 20:10:33.869076 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eae7896b21e4ffaae179e90e451137c" candidate_uuid: "4a522790944e41e4bece9ca987e64bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8648836dffb143cbb04148defddfed80" is_pre_election: true
I20260926 20:10:33.869109 25660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eae7896b21e4ffaae179e90e451137c" candidate_uuid: "4a522790944e41e4bece9ca987e64bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2427ff8243d44f7099b5d1d94db18d36" is_pre_election: true
I20260926 20:10:33.869251 25783 raft_consensus.cc:2454] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a522790944e41e4bece9ca987e64bf5 in term 0.
I20260926 20:10:33.869279 25660 raft_consensus.cc:2454] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a522790944e41e4bece9ca987e64bf5 in term 0.
I20260926 20:10:33.869632 25874 leader_election.cc:302] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a522790944e41e4bece9ca987e64bf5, 8648836dffb143cbb04148defddfed80; no voters:
I20260926 20:10:33.869918 26104 raft_consensus.cc:2781] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:33.870007 26104 raft_consensus.cc:482] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:33.870065 26104 raft_consensus.cc:3037] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:33.871134 26104 raft_consensus.cc:504] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.871562 26104 leader_election.cc:288] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [CANDIDATE]: Term 1 election: Requested vote from peers 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173), 8648836dffb143cbb04148defddfed80 (127.23.50.130:38489)
I20260926 20:10:33.871981 25783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eae7896b21e4ffaae179e90e451137c" candidate_uuid: "4a522790944e41e4bece9ca987e64bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8648836dffb143cbb04148defddfed80"
I20260926 20:10:33.871991 25660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2eae7896b21e4ffaae179e90e451137c" candidate_uuid: "4a522790944e41e4bece9ca987e64bf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2427ff8243d44f7099b5d1d94db18d36"
I20260926 20:10:33.872126 25660 raft_consensus.cc:3037] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:33.872133 25783 raft_consensus.cc:3037] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:33.873106 25783 raft_consensus.cc:2454] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a522790944e41e4bece9ca987e64bf5 in term 1.
I20260926 20:10:33.873811 25660 raft_consensus.cc:2454] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a522790944e41e4bece9ca987e64bf5 in term 1.
I20260926 20:10:33.874187 25873 leader_election.cc:302] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2427ff8243d44f7099b5d1d94db18d36, 4a522790944e41e4bece9ca987e64bf5; no voters:
I20260926 20:10:33.874642 26104 raft_consensus.cc:2781] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:33.874887 26104 raft_consensus.cc:686] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 1 LEADER]: Becoming Leader. State: Replica: 4a522790944e41e4bece9ca987e64bf5, State: Running, Role: LEADER
I20260926 20:10:33.875046 26104 consensus_queue.cc:237] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } }
I20260926 20:10:33.876957 25502 catalog_manager.cc:5495] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a522790944e41e4bece9ca987e64bf5 (127.23.50.131). New cstate: current_term: 1 leader_uuid: "4a522790944e41e4bece9ca987e64bf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a522790944e41e4bece9ca987e64bf5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37943 } health_report { overall_health: HEALTHY } } }
I20260926 20:10:33.894770 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:33.891324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.23.50.131:37943)
0926 20:10:33.891446 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:33.891450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:33.891466 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:10:33.891575 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:33.891579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:33.891632 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:33.891726 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:33.891950 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.892705 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.893597 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:33.893999 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:33.894112 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:33.894322 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:33.894406 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:33.894486 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:33.894566 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:10:33.894623 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:10:33.901575 25660 raft_consensus.cc:1261] T 2eae7896b21e4ffaae179e90e451137c P 2427ff8243d44f7099b5d1d94db18d36 [term 1 FOLLOWER]: Refusing update from remote peer 4a522790944e41e4bece9ca987e64bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:33.901597 25783 raft_consensus.cc:1261] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Refusing update from remote peer 4a522790944e41e4bece9ca987e64bf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:33.901986 26126 consensus_queue.cc:1035] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2427ff8243d44f7099b5d1d94db18d36" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 37173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:33.902136 26104 consensus_queue.cc:1035] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8648836dffb143cbb04148defddfed80" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:10:33.917833 26072 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 20:10:35.478441 25501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:35.478622 25501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:35.481858 25501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c05b50859cf64b4fb7e2ceb9fe4c8070: Sending DeleteTablet for 3 replicas of tablet 2eae7896b21e4ffaae179e90e451137c
I20260926 20:10:35.482555 25640 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eae7896b21e4ffaae179e90e451137c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:35 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:34968
I20260926 20:10:35.482589 25780 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eae7896b21e4ffaae179e90e451137c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:35 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:59140
I20260926 20:10:35.482822 25921 tablet_service.cc:1514] Processing DeleteTablet for tablet 2eae7896b21e4ffaae179e90e451137c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:35 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:33300
I20260926 20:10:35.483196 26158 tablet_replica.cc:330] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5: stopping tablet replica
I20260926 20:10:35.483386 26158 raft_consensus.cc:2229] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:10:35.483605 26158 raft_consensus.cc:2258] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:35.483695 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 25574
I20260926 20:10:35.485394 26158 ts_tablet_manager.cc:1905] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:35.490103 26157 tablet_replica.cc:330] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80: stopping tablet replica
I20260926 20:10:35.490306 26157 raft_consensus.cc:2229] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:35.490486 26157 raft_consensus.cc:2258] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:35.491252 26157 ts_tablet_manager.cc:1905] T 2eae7896b21e4ffaae179e90e451137c P 8648836dffb143cbb04148defddfed80: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 20:10:35.492110 25487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:37173 (error 108)
I20260926 20:10:35.492991 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 25715
I20260926 20:10:35.494827 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:35.492834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:37173 (local address 127.0.0.1:34988)
0926 20:10:35.494653 (+ 1819us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:35.494741 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
W20260926 20:10:35.495009 25487 catalog_manager.cc:4584] TS 2427ff8243d44f7099b5d1d94db18d36 (127.23.50.129:37173): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2eae7896b21e4ffaae179e90e451137c: Network error: Client connection negotiation failed: client connection to 127.23.50.129:37173: connect: Connection refused (error 111)
I20260926 20:10:35.542999 26158 ts_tablet_manager.cc:1918] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 20:10:35.543118 26158 log.cc:1196] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/2eae7896b21e4ffaae179e90e451137c
I20260926 20:10:35.543455 26158 ts_tablet_manager.cc:1939] T 2eae7896b21e4ffaae179e90e451137c P 4a522790944e41e4bece9ca987e64bf5: Deleting consensus metadata
I20260926 20:10:35.547519 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 25856
I20260926 20:10:35.548568 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:35.547828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:37173 (local address 127.0.0.1:34998)
0926 20:10:35.548386 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:35.548454 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442}
I20260926 20:10:35.549450 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:35.549148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:37173 (local address 127.0.0.1:35014)
0926 20:10:35.549295 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:35.549360 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:10:35.550077 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:35.549885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:38489 (local address 127.0.0.1:59154)
0926 20:10:35.549972 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:35.550021 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.130:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 20:10:35.582608 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:35.582265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:38489 (local address 127.0.0.1:59160)
0926 20:10:35.582441 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:35.582519 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.130:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 20:10:35.583580 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:35.583256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:38489 (local address 127.0.0.1:59166)
0926 20:10:35.583396 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:35.583461 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.130:38489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 20:10:35.583726 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 25470
I20260926 20:10:35.584019 26160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:35.583748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:37943 (local address 127.0.0.1:33330)
0926 20:10:35.583900 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:35.583952 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.131:37943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
2026-09-26T20:10:35Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6446 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:10:35 dist-test-slave-7034 krb5kdc[26163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:10:35 dist-test-slave-7034 krb5kdc[26163](info): set up 2 sockets
Sep 26 20:10:35 dist-test-slave-7034 krb5kdc[26163](info): commencing operation
krb5kdc: starting...
W20260926 20:10:36.950449 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.343s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:10:37 dist-test-slave-7034 krb5kdc[26163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:10:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:10:37.077071 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:39299
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:39299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:37.187796 26176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:37.188071 26176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:37.188124 26176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:37.192034 26176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:37.192087 26176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:37.192106 26176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:37.192123 26176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:37.192138 26176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:37.197288 26176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:39299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:39299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:37.198333 26176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:37.199280 26176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:37.207908 26182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:37.207995 26176 server_base.cc:1034] running on GCE node
W20260926 20:10:37.207906 26184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:37.207899 26181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:37.208765 26176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:37.209985 26176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:37.211246 26176 hybrid_clock.cc:648] HybridClock initialized: now 1790453437211228 us; error 31 us; skew 500 ppm
Sep 26 20:10:37 dist-test-slave-7034 krb5kdc[26163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453437, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:37.214445 26176 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:10:37.215421 26176 webserver.cc:460] Webserver started at http://127.23.50.190:37835/ using document root <none> and password file <none>
I20260926 20:10:37.216018 26176 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:37.216113 26176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:37.216364 26176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:37.218554 26176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "25747c9ab48d4df19f94946382ad5d73"
format_stamp: "Formatted at 2026-09-26 20:10:37 on dist-test-slave-7034"
server_key: "df74318d25ee7a33ee2b0fb48e0944d0"
server_key_iv: "7efe637f50279292d17c0f8e39d11bf2"
server_key_version: "encryptionkey@0"
I20260926 20:10:37.219167 26176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "25747c9ab48d4df19f94946382ad5d73"
format_stamp: "Formatted at 2026-09-26 20:10:37 on dist-test-slave-7034"
server_key: "df74318d25ee7a33ee2b0fb48e0944d0"
server_key_iv: "7efe637f50279292d17c0f8e39d11bf2"
server_key_version: "encryptionkey@0"
I20260926 20:10:37.223083 26176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 20:10:37.225663 26191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:37.227008 26176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:10:37.227161 26176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "25747c9ab48d4df19f94946382ad5d73"
format_stamp: "Formatted at 2026-09-26 20:10:37 on dist-test-slave-7034"
server_key: "df74318d25ee7a33ee2b0fb48e0944d0"
server_key_iv: "7efe637f50279292d17c0f8e39d11bf2"
server_key_version: "encryptionkey@0"
I20260926 20:10:37.227324 26176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:37.252218 26176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:37.255620 26176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:37.255846 26176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:37.263027 26176 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:39299
I20260926 20:10:37.263042 26244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:39299 every 8 connection(s)
I20260926 20:10:37.264068 26176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:37.266923 26245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:37.272576 26245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73: Bootstrap starting.
I20260926 20:10:37.273789 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 26176
I20260926 20:10:37.273945 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:37.274199 23754 external_mini_cluster.cc:1428] Setting key f55e1ba70fc45019c401259ea4236efa
I20260926 20:10:37.275355 26245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:37.276147 26245 log.cc:824] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:37.278568 26245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73: No bootstrap required, opened a new log
I20260926 20:10:37.283301 26245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25747c9ab48d4df19f94946382ad5d73" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39299 } }
I20260926 20:10:37.283576 26245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:37.283660 26245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25747c9ab48d4df19f94946382ad5d73, State: Initialized, Role: FOLLOWER
Sep 26 20:10:37 dist-test-slave-7034 krb5kdc[26163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:37.284219 26245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25747c9ab48d4df19f94946382ad5d73" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39299 } }
I20260926 20:10:37.284421 26245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:37.284492 26245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:37.284590 26245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:37.285763 26245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25747c9ab48d4df19f94946382ad5d73" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39299 } }
I20260926 20:10:37.286141 26245 leader_election.cc:302] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25747c9ab48d4df19f94946382ad5d73; no voters:
I20260926 20:10:37.286548 26245 leader_election.cc:288] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:37.286664 26250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:37.286908 26250 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [term 1 LEADER]: Becoming Leader. State: Replica: 25747c9ab48d4df19f94946382ad5d73, State: Running, Role: LEADER
I20260926 20:10:37.287274 26250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25747c9ab48d4df19f94946382ad5d73" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39299 } }
I20260926 20:10:37.288005 26245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:37.289001 26252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25747c9ab48d4df19f94946382ad5d73. Latest consensus state: current_term: 1 leader_uuid: "25747c9ab48d4df19f94946382ad5d73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25747c9ab48d4df19f94946382ad5d73" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39299 } } }
I20260926 20:10:37.289119 26252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:37.289515 26260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:37.289494 26251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25747c9ab48d4df19f94946382ad5d73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25747c9ab48d4df19f94946382ad5d73" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39299 } } }
I20260926 20:10:37.289659 26251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:37.290591 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:37.275345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.23.50.190:39299)
0926 20:10:37.275819 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:37.275835 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:37.275872 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:10:37.276631 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:37.276655 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:37.277078 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:37.277375 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:37.278363 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:37.279460 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:37.280705 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:37.281191 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:37.281314 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:37.284357 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:37.284363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:37.284377 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:37.284432 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:37.286288 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:37.286850 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:37.286861 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:37.286866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:37.287010 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:37.287315 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:37.287318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:37.287320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:37.287837 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:37.288048 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:37.288889 (+ 841us) server_negotiation.cc:299] Negotiation successful
0926 20:10:37.289234 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":128,"threads_started":1}
I20260926 20:10:37.292960 26260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:37.298568 26260 catalog_manager.cc:1347] Generated new cluster ID: 3da842a030624f79bce8c37ca8cae47e
I20260926 20:10:37.298642 26260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:37.313544 26260 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:37.314579 26260 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:37.320304 26260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25747c9ab48d4df19f94946382ad5d73: Generated new TSK 0
I20260926 20:10:37.321106 26260 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:10:37.353502 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39299
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:37.465297 26272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:37.465565 26272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:37.465608 26272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:37.469636 26272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:37.469695 26272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:37.469753 26272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:10:37.475045 26272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:37.476075 26272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:37.477028 26272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:37.486688 26278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:37.486696 26277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:37.486696 26280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:37.487038 26272 server_base.cc:1034] running on GCE node
I20260926 20:10:37.487427 26272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:37.488066 26272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:37.489300 26272 hybrid_clock.cc:648] HybridClock initialized: now 1790453437489271 us; error 38 us; skew 500 ppm
Sep 26 20:10:37 dist-test-slave-7034 krb5kdc[26163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453437, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:37.492410 26272 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:10:37.493345 26272 webserver.cc:460] Webserver started at http://127.23.50.129:41939/ using document root <none> and password file <none>
I20260926 20:10:37.494213 26272 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:37.494333 26272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:37.494583 26272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:37.496697 26272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "8dc285d85282421a81b34cd8333413d9"
format_stamp: "Formatted at 2026-09-26 20:10:37 on dist-test-slave-7034"
server_key: "e2d736be2d53299390d9841d82a034d8"
server_key_iv: "43606309d24b7969f095e31e461817c8"
server_key_version: "encryptionkey@0"
I20260926 20:10:37.497272 26272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "8dc285d85282421a81b34cd8333413d9"
format_stamp: "Formatted at 2026-09-26 20:10:37 on dist-test-slave-7034"
server_key: "e2d736be2d53299390d9841d82a034d8"
server_key_iv: "43606309d24b7969f095e31e461817c8"
server_key_version: "encryptionkey@0"
I20260926 20:10:37.501231 26272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:10:37.503883 26287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:37.505103 26272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:37.505249 26272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "8dc285d85282421a81b34cd8333413d9"
format_stamp: "Formatted at 2026-09-26 20:10:37 on dist-test-slave-7034"
server_key: "e2d736be2d53299390d9841d82a034d8"
server_key_iv: "43606309d24b7969f095e31e461817c8"
server_key_version: "encryptionkey@0"
I20260926 20:10:37.505374 26272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:37.523978 26272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:37.527374 26272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:37.527603 26272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:37.528254 26272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:37.529140 26272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:37.529213 26272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:37.529289 26272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:37.529336 26272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:37.539582 26272 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:42553
I20260926 20:10:37.539601 26402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:42553 every 8 connection(s)
I20260926 20:10:37.540623 26272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:10:37 dist-test-slave-7034 krb5kdc[26163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453437, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:37.550805 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 26272
I20260926 20:10:37.550920 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:10:37.551302 23754 external_mini_cluster.cc:1428] Setting key c8fd1c94077903b9baf3ae37a88a1ef2
I20260926 20:10:37.555871 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:37.542751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:36167 (local address 127.23.50.190:39299)
0926 20:10:37.542913 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:37.542917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:37.544067 (+ 1150us) server_negotiation.cc:407] Connection header received
0926 20:10:37.545031 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:37.545034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:37.545098 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:37.545207 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:37.546465 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:37.547011 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:37.548200 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:37.548457 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:37.548571 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:37.551517 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:37.551521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:37.551524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:37.551556 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:37.553417 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:37.554044 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:37.554047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:37.554049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:37.554156 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:37.554601 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:37.554604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:37.554606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:37.554905 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:37.555016 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:37.555572 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 20:10:37.555704 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:10:37.556795 26405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:37.543157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39299 (local address 127.23.50.129:36167)
0926 20:10:37.543899 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:37.543945 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:37.544816 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:37.545396 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:37.545404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:37.545775 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:37.546304 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:37.546318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:37.547149 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:37.547157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:37.548055 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:37.548064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:37.548618 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:37.548623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:37.548801 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:37.549431 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:37.549448 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:37.551241 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:37.553555 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:37.553561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:37.553580 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:37.553921 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:37.554279 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:37.554284 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:37.554287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:37.554493 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:37.555036 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:37.555041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:37.555308 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 20:10:37.555511 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 20:10:37.555888 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":131,"threads_started":1}
I20260926 20:10:37.558430 26403 heartbeater.cc:347] Connected to a master server at 127.23.50.190:39299
I20260926 20:10:37.558704 26403 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:37.559227 26403 heartbeater.cc:511] Master 127.23.50.190:39299 requested a full tablet report, sending...
I20260926 20:10:37.560807 26208 ts_manager.cc:194] Registered new tserver with Master: 8dc285d85282421a81b34cd8333413d9 (127.23.50.129:42553)
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:10:37.562086 26208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:36167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:10:37.582331 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39299
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:37.696182 26408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:37.696460 26408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:37.696503 26408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:37.700539 26408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:37.700604 26408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:37.700663 26408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:10:37.705950 26408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:37.706947 26408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:37.707967 26408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:37.717870 26414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:37.717934 26408 server_base.cc:1034] running on GCE node
W20260926 20:10:37.717890 26416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:37.717890 26413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:37.718550 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:37.719208 26408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:37.720449 26408 hybrid_clock.cc:648] HybridClock initialized: now 1790453437720430 us; error 32 us; skew 500 ppm
Sep 26 20:10:37 dist-test-slave-7034 krb5kdc[26163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453437, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:37.723508 26408 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:10:37.724429 26408 webserver.cc:460] Webserver started at http://127.23.50.130:35585/ using document root <none> and password file <none>
I20260926 20:10:37.725018 26408 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:37.725118 26408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:37.725359 26408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:37.727532 26408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "3002bdff55b94c1d94052cb3a2c0828a"
format_stamp: "Formatted at 2026-09-26 20:10:37 on dist-test-slave-7034"
server_key: "9a67d5edf73eeb1e5026e16288f7a55c"
server_key_iv: "3c97108b3841ba92c92a8ac9d56c59b5"
server_key_version: "encryptionkey@0"
I20260926 20:10:37.728116 26408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "3002bdff55b94c1d94052cb3a2c0828a"
format_stamp: "Formatted at 2026-09-26 20:10:37 on dist-test-slave-7034"
server_key: "9a67d5edf73eeb1e5026e16288f7a55c"
server_key_iv: "3c97108b3841ba92c92a8ac9d56c59b5"
server_key_version: "encryptionkey@0"
I20260926 20:10:37.732165 26408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 20:10:37.734814 26423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:37.736126 26408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:37.736307 26408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "3002bdff55b94c1d94052cb3a2c0828a"
format_stamp: "Formatted at 2026-09-26 20:10:37 on dist-test-slave-7034"
server_key: "9a67d5edf73eeb1e5026e16288f7a55c"
server_key_iv: "3c97108b3841ba92c92a8ac9d56c59b5"
server_key_version: "encryptionkey@0"
I20260926 20:10:37.736435 26408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:37.757777 26408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:37.761379 26408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:37.761607 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:37.762277 26408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:37.763171 26408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:37.763245 26408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:37.763319 26408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:37.763378 26408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:37.773135 26408 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:39221
I20260926 20:10:37.773152 26537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:39221 every 8 connection(s)
I20260926 20:10:37.774197 26408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:10:37.779160 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 26408
I20260926 20:10:37.779296 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:10:37.779577 23754 external_mini_cluster.cc:1428] Setting key b04dffc7dd14c1347a0ccb48a2dd8f76
Sep 26 20:10:37 dist-test-slave-7034 krb5kdc[26163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453437, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:10:37.789038 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:37.776240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:45025 (local address 127.23.50.190:39299)
0926 20:10:37.776419 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:37.776423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:37.777226 (+ 803us) server_negotiation.cc:407] Connection header received
0926 20:10:37.778590 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:37.778593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:37.778650 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:37.778761 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:37.780166 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:37.780698 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:37.781881 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:37.782101 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:37.782214 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:37.785160 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:37.785164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:37.785167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:37.785201 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:37.786768 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:37.787317 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:37.787322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:37.787324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:37.787409 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:37.787764 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:37.787768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:37.787770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:37.787955 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:37.788066 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:37.788746 (+ 680us) server_negotiation.cc:299] Negotiation successful
0926 20:10:37.788869 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 20:10:37.790092 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:37.776575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39299 (local address 127.23.50.130:45025)
0926 20:10:37.777067 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:37.777111 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:37.778366 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:37.778979 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:37.778987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:37.779437 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:37.780012 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:37.780026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:37.780839 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:37.780847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:37.781723 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:37.781732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:37.782256 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:37.782260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:37.782517 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:37.783174 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:37.783198 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:37.784983 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:37.786908 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:37.786914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:37.786929 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:37.787200 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:37.787510 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:37.787513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:37.787515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:37.787667 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:37.788063 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:37.788070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:37.788414 (+ 344us) client_negotiation.cc:769] Sending connection context
0926 20:10:37.788625 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 20:10:37.788964 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":128,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
I20260926 20:10:37.791411 26538 heartbeater.cc:347] Connected to a master server at 127.23.50.190:39299
I20260926 20:10:37.791666 26538 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:37.792157 26538 heartbeater.cc:511] Master 127.23.50.190:39299 requested a full tablet report, sending...
I20260926 20:10:37.793433 26208 ts_manager.cc:194] Registered new tserver with Master: 3002bdff55b94c1d94052cb3a2c0828a (127.23.50.130:39221)
I20260926 20:10:37.794106 26208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:45025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:10:37.808352 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39299
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:44701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:37.921578 26543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:37.921913 26543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:37.922008 26543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:37.926122 26543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:37.926203 26543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:37.926301 26543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:10:37.931783 26543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:37.932888 26543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:37.933954 26543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:37.943790 26549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:37.943883 26548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:37.943882 26543 server_base.cc:1034] running on GCE node
W20260926 20:10:37.944058 26551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:37.944480 26543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:37.945192 26543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:37.946416 26543 hybrid_clock.cc:648] HybridClock initialized: now 1790453437946401 us; error 31 us; skew 500 ppm
Sep 26 20:10:37 dist-test-slave-7034 krb5kdc[26163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453437, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:37.949360 26543 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:10:37.950250 26543 webserver.cc:460] Webserver started at http://127.23.50.131:35831/ using document root <none> and password file <none>
I20260926 20:10:37.950762 26543 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:37.950819 26543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:37.950977 26543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:37.952917 26543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "786d99707a6e4abc95df782fdcb75e67"
format_stamp: "Formatted at 2026-09-26 20:10:37 on dist-test-slave-7034"
server_key: "7e882db15e489865ef33e9fbf74b44ae"
server_key_iv: "95b4365a0cce3cfa418d28319a30894e"
server_key_version: "encryptionkey@0"
I20260926 20:10:37.953430 26543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "786d99707a6e4abc95df782fdcb75e67"
format_stamp: "Formatted at 2026-09-26 20:10:37 on dist-test-slave-7034"
server_key: "7e882db15e489865ef33e9fbf74b44ae"
server_key_iv: "95b4365a0cce3cfa418d28319a30894e"
server_key_version: "encryptionkey@0"
I20260926 20:10:37.957089 26543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 20:10:37.959635 26558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:37.960803 26543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:10:37.960947 26543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "786d99707a6e4abc95df782fdcb75e67"
format_stamp: "Formatted at 2026-09-26 20:10:37 on dist-test-slave-7034"
server_key: "7e882db15e489865ef33e9fbf74b44ae"
server_key_iv: "95b4365a0cce3cfa418d28319a30894e"
server_key_version: "encryptionkey@0"
I20260926 20:10:37.961062 26543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:37.978065 26543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:37.981226 26543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:37.981449 26543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:37.982129 26543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:37.982951 26543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:37.983017 26543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:37.983089 26543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:37.983130 26543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:37.991983 26543 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:43213
I20260926 20:10:37.992038 26672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:43213 every 8 connection(s)
I20260926 20:10:37.992929 26543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:10:37.994784 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 26543
I20260926 20:10:37.994926 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:10:37.995203 23754 external_mini_cluster.cc:1428] Setting key 54a2079b7462b24fc519c3d1dd616e84
Sep 26 20:10:38 dist-test-slave-7034 krb5kdc[26163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453437, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:38.006364 26248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:37.994835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:41373 (local address 127.23.50.190:39299)
0926 20:10:37.994976 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:37.994980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:37.996169 (+ 1189us) server_negotiation.cc:407] Connection header received
0926 20:10:37.997077 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:37.997079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:37.997138 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:37.997224 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:37.998427 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:37.998954 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:38.000083 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:38.000313 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:38.000396 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:38.002840 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:38.002843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:38.002845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:38.002870 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:38.004411 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:38.004884 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:38.004887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:38.004888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:38.004965 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:38.005308 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:38.005311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:38.005312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:38.005484 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:38.005561 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:38.006107 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 20:10:38.006210 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:10:38.007162 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:37.995247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39299 (local address 127.23.50.131:41373)
0926 20:10:37.996015 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:37.996074 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:37.996888 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:37.997446 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:37.997453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:37.997803 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:37.998305 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:37.998318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:37.999067 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:37.999070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:37.999967 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:37.999976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:38.000442 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:38.000445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:38.000580 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:38.001140 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:38.001167 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:38.002678 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:38.004518 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:38.004522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:38.004534 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:38.004779 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:38.005047 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:38.005049 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:38.005050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:38.005184 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:38.005572 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:38.005576 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:38.005829 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 20:10:38.006061 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 20:10:38.006321 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":136,"threads_started":1}
I20260926 20:10:38.008585 26673 heartbeater.cc:347] Connected to a master server at 127.23.50.190:39299
I20260926 20:10:38.008857 26673 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:38.009438 26673 heartbeater.cc:511] Master 127.23.50.190:39299 requested a full tablet report, sending...
I20260926 20:10:38.010432 26208 ts_manager.cc:194] Registered new tserver with Master: 786d99707a6e4abc95df782fdcb75e67 (127.23.50.131:43213)
I20260926 20:10:38.010993 26208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:41373
I20260926 20:10:38.020792 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:10:38.034456 26248 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:10:38.032905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.23.50.190:39299)
0926 20:10:38.033075 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:38.033079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:38.033165 (+ 86us) server_negotiation.cc:407] Connection header received
0926 20:10:38.033356 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:38.033359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:38.033408 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:38.033483 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:38.034301 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60344: BlockingRecv error: recv got EOF from 127.0.0.1:60344 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:10:38.035475 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 26272
I20260926 20:10:38.041049 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 26408
I20260926 20:10:38.046142 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 26543
I20260926 20:10:38.051368 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 26176
2026-09-26T20:10:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2465 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:10:38 dist-test-slave-7034 krb5kdc[26685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:10:38 dist-test-slave-7034 krb5kdc[26685](info): set up 2 sockets
Sep 26 20:10:38 dist-test-slave-7034 krb5kdc[26685](info): commencing operation
krb5kdc: starting...
W20260926 20:10:39.098446 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:10:39 dist-test-slave-7034 krb5kdc[26685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:10:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:10:39.220994 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:35885
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:35885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:39.331487 26698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:39.331743 26698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:39.331786 26698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:39.335686 26698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:39.335745 26698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:39.335765 26698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:39.335783 26698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:39.335810 26698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:39.341089 26698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:35885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:35885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:39.342141 26698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:39.343102 26698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:39.351897 26703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:39.351882 26704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:39.351945 26698 server_base.cc:1034] running on GCE node
W20260926 20:10:39.351889 26706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:39.352705 26698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:39.353996 26698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:39.355242 26698 hybrid_clock.cc:648] HybridClock initialized: now 1790453439355216 us; error 55 us; skew 500 ppm
Sep 26 20:10:39 dist-test-slave-7034 krb5kdc[26685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453439, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:39.361697 26698 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:10:39.362751 26698 webserver.cc:460] Webserver started at http://127.23.50.190:36673/ using document root <none> and password file <none>
I20260926 20:10:39.363303 26698 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:39.363364 26698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:39.363533 26698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:39.365608 26698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "785a93a53b6142d0a416da72a83ad403"
format_stamp: "Formatted at 2026-09-26 20:10:39 on dist-test-slave-7034"
server_key: "8f3ee3dbe339aa960d186cd0dc586719"
server_key_iv: "ba3cf5206c6cfd9228c2d5cc4be327f8"
server_key_version: "encryptionkey@0"
I20260926 20:10:39.366206 26698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "785a93a53b6142d0a416da72a83ad403"
format_stamp: "Formatted at 2026-09-26 20:10:39 on dist-test-slave-7034"
server_key: "8f3ee3dbe339aa960d186cd0dc586719"
server_key_iv: "ba3cf5206c6cfd9228c2d5cc4be327f8"
server_key_version: "encryptionkey@0"
I20260926 20:10:39.370023 26698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:10:39.372521 26713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:39.373930 26698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:39.374038 26698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "785a93a53b6142d0a416da72a83ad403"
format_stamp: "Formatted at 2026-09-26 20:10:39 on dist-test-slave-7034"
server_key: "8f3ee3dbe339aa960d186cd0dc586719"
server_key_iv: "ba3cf5206c6cfd9228c2d5cc4be327f8"
server_key_version: "encryptionkey@0"
I20260926 20:10:39.374130 26698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:39.386590 26698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:39.389497 26698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:39.389645 26698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:39.396536 26698 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:35885
I20260926 20:10:39.396838 26766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:35885 every 8 connection(s)
I20260926 20:10:39.397524 26698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:39.400362 26767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:39.405788 26767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403: Bootstrap starting.
I20260926 20:10:39.407660 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 26698
I20260926 20:10:39.407797 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:39.408051 23754 external_mini_cluster.cc:1428] Setting key a514c9f1c91380bc273246faf6724d33
I20260926 20:10:39.408381 26767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:39.409294 26767 log.cc:824] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:39.411410 26767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403: No bootstrap required, opened a new log
I20260926 20:10:39.415055 26767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785a93a53b6142d0a416da72a83ad403" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35885 } }
I20260926 20:10:39.415390 26767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:39.415486 26767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 785a93a53b6142d0a416da72a83ad403, State: Initialized, Role: FOLLOWER
I20260926 20:10:39.415936 26767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785a93a53b6142d0a416da72a83ad403" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35885 } }
I20260926 20:10:39.416088 26767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:39.416175 26767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:39.416304 26767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 20:10:39 dist-test-slave-7034 krb5kdc[26685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:39.417326 26767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785a93a53b6142d0a416da72a83ad403" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35885 } }
I20260926 20:10:39.417689 26767 leader_election.cc:302] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 785a93a53b6142d0a416da72a83ad403; no voters:
I20260926 20:10:39.418138 26767 leader_election.cc:288] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:39.418481 26772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:39.418669 26772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [term 1 LEADER]: Becoming Leader. State: Replica: 785a93a53b6142d0a416da72a83ad403, State: Running, Role: LEADER
I20260926 20:10:39.418970 26772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785a93a53b6142d0a416da72a83ad403" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35885 } }
I20260926 20:10:39.419498 26767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:39.420579 26774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 785a93a53b6142d0a416da72a83ad403. Latest consensus state: current_term: 1 leader_uuid: "785a93a53b6142d0a416da72a83ad403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785a93a53b6142d0a416da72a83ad403" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35885 } } }
I20260926 20:10:39.420699 26774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:39.421000 26773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "785a93a53b6142d0a416da72a83ad403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "785a93a53b6142d0a416da72a83ad403" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35885 } } }
I20260926 20:10:39.421109 26773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:39.421211 26781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:39.423363 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:39.409140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.23.50.190:35885)
0926 20:10:39.409614 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:39.409629 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:39.409692 (+ 63us) server_negotiation.cc:407] Connection header received
0926 20:10:39.410486 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:39.410505 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:39.410849 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:39.411191 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:39.412043 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:39.413034 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:39.413978 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:39.414465 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:39.414634 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:39.417356 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:39.417361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:39.417375 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:39.417402 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:39.419413 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:39.419888 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:39.419894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:39.419900 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:39.420056 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:39.420444 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:39.420449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:39.420452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:39.421022 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:39.421222 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:39.422129 (+ 907us) server_negotiation.cc:299] Negotiation successful
0926 20:10:39.422432 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
I20260926 20:10:39.425830 26781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:39.426451 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:10:39.431449 26781 catalog_manager.cc:1347] Generated new cluster ID: 4f5a3c6655ce44648874ab1c612b69f2
I20260926 20:10:39.431505 26781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:39.446281 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:39.440568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40952 (local address 127.23.50.190:35885)
0926 20:10:39.440674 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:39.440678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:39.440957 (+ 279us) server_negotiation.cc:407] Connection header received
0926 20:10:39.441065 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:39.441068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:39.441122 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:39.441208 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:39.441760 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:39.442606 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:39.443778 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:39.444169 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:39.444274 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:39.445167 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:39.445171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:39.445174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:39.445789 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:39.445824 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:39.445870 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:39.445952 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:39.446064 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 20:10:39.446128 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:10:39.452771 26781 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:39.454033 26781 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:39.464716 26781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 785a93a53b6142d0a416da72a83ad403: Generated new TSK 0
I20260926 20:10:39.465503 26781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:10:39.475689 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 26698
2026-09-26T20:10:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1422 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:10:39 dist-test-slave-7034 krb5kdc[26800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:10:39 dist-test-slave-7034 krb5kdc[26800](info): set up 2 sockets
Sep 26 20:10:39 dist-test-slave-7034 krb5kdc[26800](info): commencing operation
krb5kdc: starting...
W20260926 20:10:40.502632 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:10:40 dist-test-slave-7034 krb5kdc[26800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:10:40.628643 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:43439
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:43439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:40.741397 26813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:40.741699 26813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:40.741743 26813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:40.745976 26813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:40.746048 26813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:40.746068 26813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:40.746084 26813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:40.746100 26813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:40.751652 26813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:43439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:43439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:40.752827 26813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:40.753975 26813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:40.763056 26819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:40.763103 26813 server_base.cc:1034] running on GCE node
W20260926 20:10:40.763252 26821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:40.763103 26818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:40.763962 26813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:40.765151 26813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:40.766324 26813 hybrid_clock.cc:648] HybridClock initialized: now 1790453440766302 us; error 37 us; skew 500 ppm
Sep 26 20:10:40 dist-test-slave-7034 krb5kdc[26800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453440, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:40.769176 26813 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:10:40.770141 26813 webserver.cc:460] Webserver started at http://127.23.50.190:35095/ using document root <none> and password file <none>
I20260926 20:10:40.770677 26813 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:40.770737 26813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:40.770920 26813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:40.772961 26813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "5842017eba824e3dbdd9cdbcc8d3e20b"
format_stamp: "Formatted at 2026-09-26 20:10:40 on dist-test-slave-7034"
server_key: "81667eaa50099437d7badce6b610817f"
server_key_iv: "1d932dd182c4ab8dba1a0557a016fbe8"
server_key_version: "encryptionkey@0"
I20260926 20:10:40.773470 26813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "5842017eba824e3dbdd9cdbcc8d3e20b"
format_stamp: "Formatted at 2026-09-26 20:10:40 on dist-test-slave-7034"
server_key: "81667eaa50099437d7badce6b610817f"
server_key_iv: "1d932dd182c4ab8dba1a0557a016fbe8"
server_key_version: "encryptionkey@0"
I20260926 20:10:40.777304 26813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:10:40.779816 26828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:40.781085 26813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:40.781198 26813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "5842017eba824e3dbdd9cdbcc8d3e20b"
format_stamp: "Formatted at 2026-09-26 20:10:40 on dist-test-slave-7034"
server_key: "81667eaa50099437d7badce6b610817f"
server_key_iv: "1d932dd182c4ab8dba1a0557a016fbe8"
server_key_version: "encryptionkey@0"
I20260926 20:10:40.781291 26813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:40.799607 26813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:40.803006 26813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:40.803164 26813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:40.810439 26813 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:43439
I20260926 20:10:40.810827 26881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:43439 every 8 connection(s)
I20260926 20:10:40.811493 26813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:40.814659 26882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:40.815251 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 26813
I20260926 20:10:40.815394 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:40.815732 23754 external_mini_cluster.cc:1428] Setting key ab4c54807a23be1dfd90f6cc9c3aab55
I20260926 20:10:40.822888 26882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b: Bootstrap starting.
I20260926 20:10:40.825528 26882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b: Neither blocks nor log segments found. Creating new log.
Sep 26 20:10:40 dist-test-slave-7034 krb5kdc[26800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:40.826512 26882 log.cc:824] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:40.828939 26882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b: No bootstrap required, opened a new log
I20260926 20:10:40.831578 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:40.817150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.23.50.190:43439)
0926 20:10:40.817645 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:40.817667 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:40.817716 (+ 49us) server_negotiation.cc:407] Connection header received
0926 20:10:40.818623 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:40.818649 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:40.819121 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:40.819532 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:40.820226 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:40.821932 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:40.822807 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:40.823542 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:40.823736 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:40.826295 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:40.826301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:40.826320 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:40.826358 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:40.828351 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:40.828918 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:40.828924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:40.828929 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:40.829144 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:40.829492 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:40.829495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:40.829496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:40.830040 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:40.830212 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:40.830478 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 20:10:40.830827 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":162,"threads_started":1}
I20260926 20:10:40.833050 26882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5842017eba824e3dbdd9cdbcc8d3e20b" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43439 } }
I20260926 20:10:40.833325 26882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:40.833381 26882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5842017eba824e3dbdd9cdbcc8d3e20b, State: Initialized, Role: FOLLOWER
I20260926 20:10:40.833827 26882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5842017eba824e3dbdd9cdbcc8d3e20b" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43439 } }
I20260926 20:10:40.834040 26882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:40.834113 26882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:40.834216 26882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:40.835146 26882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5842017eba824e3dbdd9cdbcc8d3e20b" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43439 } }
I20260926 20:10:40.835490 26882 leader_election.cc:302] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5842017eba824e3dbdd9cdbcc8d3e20b; no voters:
I20260926 20:10:40.835819 26882 leader_election.cc:288] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:40.835940 26887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:40.836527 26887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [term 1 LEADER]: Becoming Leader. State: Replica: 5842017eba824e3dbdd9cdbcc8d3e20b, State: Running, Role: LEADER
I20260926 20:10:40.836880 26887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5842017eba824e3dbdd9cdbcc8d3e20b" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43439 } }
I20260926 20:10:40.837088 26882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:40.838450 26889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5842017eba824e3dbdd9cdbcc8d3e20b. Latest consensus state: current_term: 1 leader_uuid: "5842017eba824e3dbdd9cdbcc8d3e20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5842017eba824e3dbdd9cdbcc8d3e20b" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43439 } } }
I20260926 20:10:40.838585 26889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:40.838959 26898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:40.838917 26888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5842017eba824e3dbdd9cdbcc8d3e20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5842017eba824e3dbdd9cdbcc8d3e20b" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43439 } } }
I20260926 20:10:40.839001 26888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:40.841578 26898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:40.847611 26898 catalog_manager.cc:1347] Generated new cluster ID: 8139f30a9e2b4ac587ff6dbf1126b88a
I20260926 20:10:40.847681 26898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:40.868032 26898 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:40.869195 26898 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:40.881212 26898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5842017eba824e3dbdd9cdbcc8d3e20b: Generated new TSK 0
I20260926 20:10:40.882115 26898 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:10:40.918103 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43439
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:41.033433 26908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:41.033762 26908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:41.033838 26908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:41.038024 26908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:41.038116 26908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:41.038224 26908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:10:41.043776 26908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:41.044929 26908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:41.046079 26908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:41.056879 26913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:41.057400 26914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:41.057371 26908 server_base.cc:1034] running on GCE node
W20260926 20:10:41.056879 26916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:41.057940 26908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:41.058722 26908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:41.059859 26908 hybrid_clock.cc:648] HybridClock initialized: now 1790453441059832 us; error 48 us; skew 500 ppm
Sep 26 20:10:41 dist-test-slave-7034 krb5kdc[26800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:41.063961 26908 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:10:41.065114 26908 webserver.cc:460] Webserver started at http://127.23.50.129:45607/ using document root <none> and password file <none>
I20260926 20:10:41.065796 26908 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:41.065937 26908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:41.066181 26908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:41.068509 26908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "c1a0b81c9f5943e5a947c0678680b60b"
format_stamp: "Formatted at 2026-09-26 20:10:41 on dist-test-slave-7034"
server_key: "1ae2bc09da55af6b049b11893baf5861"
server_key_iv: "7a28d5a8920f387baf7cf9f2a2c0a4be"
server_key_version: "encryptionkey@0"
I20260926 20:10:41.069160 26908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "c1a0b81c9f5943e5a947c0678680b60b"
format_stamp: "Formatted at 2026-09-26 20:10:41 on dist-test-slave-7034"
server_key: "1ae2bc09da55af6b049b11893baf5861"
server_key_iv: "7a28d5a8920f387baf7cf9f2a2c0a4be"
server_key_version: "encryptionkey@0"
I20260926 20:10:41.073243 26908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:10:41.075932 26923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:41.077306 26908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:41.077447 26908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "c1a0b81c9f5943e5a947c0678680b60b"
format_stamp: "Formatted at 2026-09-26 20:10:41 on dist-test-slave-7034"
server_key: "1ae2bc09da55af6b049b11893baf5861"
server_key_iv: "7a28d5a8920f387baf7cf9f2a2c0a4be"
server_key_version: "encryptionkey@0"
I20260926 20:10:41.077565 26908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:41.091434 26908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:41.095502 26908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:41.095741 26908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:41.096473 26908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:41.097514 26908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:41.097590 26908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:41.097638 26908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:41.097699 26908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:41.109076 26908 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:35017
I20260926 20:10:41.109166 27037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:35017 every 8 connection(s)
I20260926 20:10:41.110216 26908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:10:41.115633 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 26908
I20260926 20:10:41.115761 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:10:41.116089 23754 external_mini_cluster.cc:1428] Setting key 30c89623f07f85412eb13ba31185724b
Sep 26 20:10:41 dist-test-slave-7034 krb5kdc[26800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:41.126154 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:41.112267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:39877 (local address 127.23.50.190:43439)
0926 20:10:41.112424 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:41.112428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:41.113354 (+ 926us) server_negotiation.cc:407] Connection header received
0926 20:10:41.114847 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:41.114851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:41.114911 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:41.115020 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:41.116429 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.116975 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.118241 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.118494 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.118608 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:41.121570 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:41.121577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:41.121583 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:41.121619 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:41.123661 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:41.124277 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:41.124282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:41.124291 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:41.124398 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:41.124791 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:41.124797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:41.124800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:41.125069 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:41.125180 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:41.125760 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 20:10:41.125951 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:10:41.126996 27040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:41.112675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43439 (local address 127.23.50.129:39877)
0926 20:10:41.113184 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:41.113228 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:41.114607 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:41.115212 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:41.115220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:41.115604 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:41.116269 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:41.116286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.117111 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.117115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:41.118109 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:41.118120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.118664 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.118669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:41.118850 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:41.119547 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:41.119572 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:41.121377 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:41.123809 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:41.123814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:41.123829 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:41.124116 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:41.124527 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:41.124529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:41.124531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:41.124686 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:41.125227 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:41.125233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:41.125504 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 20:10:41.125677 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 20:10:41.126033 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":123,"threads_started":1}
I20260926 20:10:41.128376 27038 heartbeater.cc:347] Connected to a master server at 127.23.50.190:43439
I20260926 20:10:41.128607 27038 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:41.129086 27038 heartbeater.cc:511] Master 127.23.50.190:43439 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
I20260926 20:10:41.130911 26845 ts_manager.cc:194] Registered new tserver with Master: c1a0b81c9f5943e5a947c0678680b60b (127.23.50.129:35017)
I20260926 20:10:41.132450 26845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:39877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:10:41.150278 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43439
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:41.265287 27043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:41.265712 27043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:41.265829 27043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:41.270814 27043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:41.270881 27043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:41.270942 27043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:10:41.276495 27043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:41.277577 27043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:41.278647 27043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:41.288691 27051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:41.288828 27048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:41.288700 27049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:41.289062 27043 server_base.cc:1034] running on GCE node
I20260926 20:10:41.289579 27043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:41.290297 27043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:41.291502 27043 hybrid_clock.cc:648] HybridClock initialized: now 1790453441291478 us; error 45 us; skew 500 ppm
Sep 26 20:10:41 dist-test-slave-7034 krb5kdc[26800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:41.294478 27043 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:10:41.295538 27043 webserver.cc:460] Webserver started at http://127.23.50.130:40253/ using document root <none> and password file <none>
I20260926 20:10:41.296142 27043 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:41.296248 27043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:41.296499 27043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:41.298810 27043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "1ad3e1738a724faa9b7e16f3b5637d61"
format_stamp: "Formatted at 2026-09-26 20:10:41 on dist-test-slave-7034"
server_key: "25cc3b527641f45b60157f6860e77feb"
server_key_iv: "112bb387c40a0632f7f4d7af7bf36b1d"
server_key_version: "encryptionkey@0"
I20260926 20:10:41.299430 27043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "1ad3e1738a724faa9b7e16f3b5637d61"
format_stamp: "Formatted at 2026-09-26 20:10:41 on dist-test-slave-7034"
server_key: "25cc3b527641f45b60157f6860e77feb"
server_key_iv: "112bb387c40a0632f7f4d7af7bf36b1d"
server_key_version: "encryptionkey@0"
I20260926 20:10:41.303411 27043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:10:41.306077 27058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:41.307318 27043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:10:41.307451 27043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "1ad3e1738a724faa9b7e16f3b5637d61"
format_stamp: "Formatted at 2026-09-26 20:10:41 on dist-test-slave-7034"
server_key: "25cc3b527641f45b60157f6860e77feb"
server_key_iv: "112bb387c40a0632f7f4d7af7bf36b1d"
server_key_version: "encryptionkey@0"
I20260926 20:10:41.307562 27043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:41.322747 27043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:41.325627 27043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:41.325826 27043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:41.326454 27043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:41.327250 27043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:41.327320 27043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:41.327396 27043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:41.327458 27043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:41.336457 27043 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:45287
I20260926 20:10:41.336562 27173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:45287 every 8 connection(s)
I20260926 20:10:41.337448 27043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:10:41.346823 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 27043
I20260926 20:10:41.346969 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:10:41.347255 23754 external_mini_cluster.cc:1428] Setting key 0fe611785c6bde714a3f55424acd55c1
Sep 26 20:10:41 dist-test-slave-7034 krb5kdc[26800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:41.352094 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:41.339436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:35293 (local address 127.23.50.190:43439)
0926 20:10:41.339580 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:41.339584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:41.340563 (+ 979us) server_negotiation.cc:407] Connection header received
0926 20:10:41.341999 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:41.342002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:41.342059 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:41.342139 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:41.343309 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.343851 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.345010 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.345254 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.345372 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:41.348141 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:41.348145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:41.348148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:41.348221 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:41.349810 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:41.350454 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:41.350457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:41.350459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:41.350540 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:41.350950 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:41.350953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:41.350954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:41.351136 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:41.351216 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:41.351867 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 20:10:41.351976 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:10:41.353358 27176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:41.339933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43439 (local address 127.23.50.130:35293)
0926 20:10:41.340401 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:41.340445 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:41.341723 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:41.342308 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:41.342314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:41.342682 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:41.343154 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:41.343170 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.343992 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.343996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:41.344878 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:41.344887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.345410 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.345414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:41.345589 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:41.346235 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:41.346251 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:41.347946 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:41.349977 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:41.349983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:41.349999 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:41.350332 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:41.350637 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:41.350641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:41.350644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:41.350838 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:41.351243 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:41.351252 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:41.351592 (+ 340us) client_negotiation.cc:769] Sending connection context
0926 20:10:41.351809 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 20:10:41.352191 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260926 20:10:41.354648 27174 heartbeater.cc:347] Connected to a master server at 127.23.50.190:43439
I20260926 20:10:41.354924 27174 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:41.355434 27174 heartbeater.cc:511] Master 127.23.50.190:43439 requested a full tablet report, sending...
I20260926 20:10:41.356560 26845 ts_manager.cc:194] Registered new tserver with Master: 1ad3e1738a724faa9b7e16f3b5637d61 (127.23.50.130:45287)
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:10:41.357388 26845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:35293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:10:41.377032 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43439
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:45079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:41.494484 27179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:41.494760 27179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:41.494803 27179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:41.498986 27179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:41.499043 27179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:41.499101 27179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:10:41.504462 27179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:45079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:41.505429 27179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:41.506407 27179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:41.516045 27187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:41.516026 27184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:41.516031 27185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:41.516538 27179 server_base.cc:1034] running on GCE node
I20260926 20:10:41.516971 27179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:41.517634 27179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:41.518860 27179 hybrid_clock.cc:648] HybridClock initialized: now 1790453441518841 us; error 69 us; skew 500 ppm
Sep 26 20:10:41 dist-test-slave-7034 krb5kdc[26800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:41.521752 27179 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:10:41.522713 27179 webserver.cc:460] Webserver started at http://127.23.50.131:45351/ using document root <none> and password file <none>
I20260926 20:10:41.523309 27179 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:41.523381 27179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:41.523554 27179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:41.525624 27179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "d61b77ce89004c4d8bcf303fbb33879e"
format_stamp: "Formatted at 2026-09-26 20:10:41 on dist-test-slave-7034"
server_key: "cc1b6d7bfa9bcbf85f374ed4517cd29b"
server_key_iv: "0cc0158731519767aa56e4568fd9862d"
server_key_version: "encryptionkey@0"
I20260926 20:10:41.526254 27179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "d61b77ce89004c4d8bcf303fbb33879e"
format_stamp: "Formatted at 2026-09-26 20:10:41 on dist-test-slave-7034"
server_key: "cc1b6d7bfa9bcbf85f374ed4517cd29b"
server_key_iv: "0cc0158731519767aa56e4568fd9862d"
server_key_version: "encryptionkey@0"
I20260926 20:10:41.530690 27179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:10:41.533186 27194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:41.534624 27179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:41.534754 27179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "d61b77ce89004c4d8bcf303fbb33879e"
format_stamp: "Formatted at 2026-09-26 20:10:41 on dist-test-slave-7034"
server_key: "cc1b6d7bfa9bcbf85f374ed4517cd29b"
server_key_iv: "0cc0158731519767aa56e4568fd9862d"
server_key_version: "encryptionkey@0"
I20260926 20:10:41.534868 27179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:41.569334 27179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:41.572849 27179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:41.573062 27179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:41.573741 27179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:41.574608 27179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:41.574656 27179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:41.574725 27179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:41.574756 27179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:41.583338 27179 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:46321
I20260926 20:10:41.583422 27307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:46321 every 8 connection(s)
I20260926 20:10:41.584326 27179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:10:41.594149 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 27179
Sep 26 20:10:41 dist-test-slave-7034 krb5kdc[26800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:41.594249 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:10:41.594527 23754 external_mini_cluster.cc:1428] Setting key e6314751d0b1e1d2751d64fe7b56f8b1
I20260926 20:10:41.598752 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:41.586391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:60413 (local address 127.23.50.190:43439)
0926 20:10:41.586524 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:41.586528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:41.587454 (+ 926us) server_negotiation.cc:407] Connection header received
0926 20:10:41.588887 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:41.588890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:41.588940 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:41.589014 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:41.590209 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.590785 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.591936 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.592155 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.592236 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:41.594943 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:41.594946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:41.594948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:41.594972 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:41.596594 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:41.597156 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:41.597160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:41.597162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:41.597252 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:41.597615 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:41.597627 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:41.597629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:41.597823 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:41.597959 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:41.598470 (+ 511us) server_negotiation.cc:299] Negotiation successful
0926 20:10:41.598594 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:10:41.599587 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:41.586745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43439 (local address 127.23.50.131:60413)
0926 20:10:41.587294 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:41.587337 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:41.588687 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:41.589154 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:41.589160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:41.589508 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:41.590043 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:41.590058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.590904 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.590906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:41.591821 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:41.591830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.592289 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.592292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:41.592428 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:41.593002 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:41.593017 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:41.594754 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:41.596740 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:41.596746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:41.596763 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:41.597046 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:41.597356 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:41.597360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:41.597362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:41.597517 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:41.597979 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:41.597986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:41.598233 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 20:10:41.598429 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 20:10:41.598762 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":118,"threads_started":1}
I20260926 20:10:41.600708 27308 heartbeater.cc:347] Connected to a master server at 127.23.50.190:43439
I20260926 20:10:41.601012 27308 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:41.601516 27308 heartbeater.cc:511] Master 127.23.50.190:43439 requested a full tablet report, sending...
I20260926 20:10:41.602568 26845 ts_manager.cc:194] Registered new tserver with Master: d61b77ce89004c4d8bcf303fbb33879e (127.23.50.131:46321)
I20260926 20:10:41.603148 26845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:60413
I20260926 20:10:41.608522 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:10:41 dist-test-slave-7034 krb5kdc[26800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453441, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 20:10:41 dist-test-slave-7034 krb5kdc[26800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453441, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:41.634239 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:41.625589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.23.50.190:43439)
0926 20:10:41.625749 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:41.625754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:41.625889 (+ 135us) server_negotiation.cc:407] Connection header received
0926 20:10:41.626073 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:41.626075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:41.626119 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:41.626204 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:41.626944 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.627467 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.628345 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:41.628571 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:41.628682 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:41.631294 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:41.631298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:41.631300 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:41.631328 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:41.632620 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:41.633024 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:41.633027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:41.633028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:41.633102 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:41.633367 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:41.633369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:41.633370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:41.633592 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:41.633693 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:41.633973 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 20:10:41.634074 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 20:10:41.634663 26845 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33848
I20260926 20:10:41.635558 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 26908
I20260926 20:10:41.641105 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 27043
I20260926 20:10:41.646325 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 27179
I20260926 20:10:41.651480 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 26813
2026-09-26T20:10:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2178 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:10:41 dist-test-slave-7034 krb5kdc[27321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:10:41 dist-test-slave-7034 krb5kdc[27321](info): set up 2 sockets
Sep 26 20:10:41 dist-test-slave-7034 krb5kdc[27321](info): commencing operation
krb5kdc: starting...
W20260926 20:10:42.703060 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:10:42 dist-test-slave-7034 krb5kdc[27321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:10:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:10:42.832902 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37403
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:37403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:42.944931 27334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:42.945230 27334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:42.945274 27334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:42.949479 27334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:42.949556 27334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:42.949577 27334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:42.949594 27334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:42.949610 27334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:42.955185 27334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:37403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:42.956276 27334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:42.957239 27334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:42.966178 27339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:42.966178 27340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:42.966261 27342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:42.966368 27334 server_base.cc:1034] running on GCE node
I20260926 20:10:42.967028 27334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:42.968336 27334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:42.969561 27334 hybrid_clock.cc:648] HybridClock initialized: now 1790453442969532 us; error 41 us; skew 500 ppm
Sep 26 20:10:42 dist-test-slave-7034 krb5kdc[27321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453442, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:42.972592 27334 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:10:42.973555 27334 webserver.cc:460] Webserver started at http://127.23.50.190:37529/ using document root <none> and password file <none>
I20260926 20:10:42.974172 27334 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:42.974238 27334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:42.974411 27334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:42.976434 27334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "f7e09532500e42069c57ca202f7b7c51"
format_stamp: "Formatted at 2026-09-26 20:10:42 on dist-test-slave-7034"
server_key: "e3c14de6b052bc74a186bda382c8f7c5"
server_key_iv: "03383416862dcf307b352c6e0ff691db"
server_key_version: "encryptionkey@0"
I20260926 20:10:42.976949 27334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "f7e09532500e42069c57ca202f7b7c51"
format_stamp: "Formatted at 2026-09-26 20:10:42 on dist-test-slave-7034"
server_key: "e3c14de6b052bc74a186bda382c8f7c5"
server_key_iv: "03383416862dcf307b352c6e0ff691db"
server_key_version: "encryptionkey@0"
I20260926 20:10:42.981690 27334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 20:10:42.984362 27349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:42.985656 27334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:42.985764 27334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "f7e09532500e42069c57ca202f7b7c51"
format_stamp: "Formatted at 2026-09-26 20:10:42 on dist-test-slave-7034"
server_key: "e3c14de6b052bc74a186bda382c8f7c5"
server_key_iv: "03383416862dcf307b352c6e0ff691db"
server_key_version: "encryptionkey@0"
I20260926 20:10:42.985934 27334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:43.002130 27334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:43.005128 27334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:43.005292 27334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:43.012694 27334 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:37403
I20260926 20:10:43.012725 27401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:37403 every 8 connection(s)
I20260926 20:10:43.013644 27334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:43.016445 27402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:43.019306 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 27334
I20260926 20:10:43.019436 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:43.019740 23754 external_mini_cluster.cc:1428] Setting key c9eb67cc9a78965e8bac9789a8e2ddef
I20260926 20:10:43.022759 27402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51: Bootstrap starting.
I20260926 20:10:43.024899 27402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:43.025656 27402 log.cc:824] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:43.027714 27402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51: No bootstrap required, opened a new log
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:43.031610 27402 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e09532500e42069c57ca202f7b7c51" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37403 } }
I20260926 20:10:43.031864 27402 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:43.031960 27402 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e09532500e42069c57ca202f7b7c51, State: Initialized, Role: FOLLOWER
I20260926 20:10:43.032451 27402 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e09532500e42069c57ca202f7b7c51" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37403 } }
I20260926 20:10:43.032610 27402 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:43.032698 27402 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:43.032804 27402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:43.033888 27402 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e09532500e42069c57ca202f7b7c51" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37403 } }
I20260926 20:10:43.034256 27402 leader_election.cc:302] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7e09532500e42069c57ca202f7b7c51; no voters:
I20260926 20:10:43.034575 27402 leader_election.cc:288] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:43.034633 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:43.021192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.23.50.190:37403)
0926 20:10:43.021641 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:43.021656 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:43.021692 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:10:43.022411 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:43.022431 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:43.022765 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:43.023153 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:43.023944 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.024930 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.025924 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.026600 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.026774 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:43.029417 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:43.029422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:43.029435 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:43.029462 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:43.031356 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:43.031847 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:43.031853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:43.031857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:43.032032 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:43.032355 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:43.032358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:43.032360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:43.032870 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:43.033073 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:43.033379 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 20:10:43.033671 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":129,"threads_started":1}
I20260926 20:10:43.034704 27407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:43.035248 27407 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [term 1 LEADER]: Becoming Leader. State: Replica: f7e09532500e42069c57ca202f7b7c51, State: Running, Role: LEADER
I20260926 20:10:43.035542 27407 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e09532500e42069c57ca202f7b7c51" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37403 } }
I20260926 20:10:43.035880 27402 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:43.037698 27409 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7e09532500e42069c57ca202f7b7c51. Latest consensus state: current_term: 1 leader_uuid: "f7e09532500e42069c57ca202f7b7c51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e09532500e42069c57ca202f7b7c51" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37403 } } }
I20260926 20:10:43.037695 27408 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7e09532500e42069c57ca202f7b7c51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e09532500e42069c57ca202f7b7c51" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37403 } } }
I20260926 20:10:43.037870 27409 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:43.037870 27408 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51 [sys.catalog]: This master's current role is: LEADER
W20260926 20:10:43.040740 27421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:10:43.040818 27421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:10:43.040897 27422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:43.041611 27422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:43.047465 27422 catalog_manager.cc:1347] Generated new cluster ID: a920c89e08f44ebb8d5614912ad8f4c4
I20260926 20:10:43.047542 27422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:43.067061 27422 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:43.068068 27422 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:43.076869 27422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7e09532500e42069c57ca202f7b7c51: Generated new TSK 0
I20260926 20:10:43.077566 27422 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:10:43.109084 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37403
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:43.226119 27428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:43.226403 27428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:43.226447 27428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:43.230687 27428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:43.230763 27428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:43.230825 27428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:10:43.236474 27428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:43.237746 27428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:43.238909 27428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:43.249042 27434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:43.249028 27433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:43.249174 27428 server_base.cc:1034] running on GCE node
W20260926 20:10:43.249028 27436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:43.249804 27428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:43.250524 27428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:43.251726 27428 hybrid_clock.cc:648] HybridClock initialized: now 1790453443251701 us; error 41 us; skew 500 ppm
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453443, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:43.254685 27428 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:10:43.255730 27428 webserver.cc:460] Webserver started at http://127.23.50.129:41835/ using document root <none> and password file <none>
I20260926 20:10:43.256318 27428 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:43.256410 27428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:43.256649 27428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:43.259058 27428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "420020f3337c4b6e925f585db59b9993"
format_stamp: "Formatted at 2026-09-26 20:10:43 on dist-test-slave-7034"
server_key: "82cd78b8e521e43bb1ba42eb253c285b"
server_key_iv: "2a589dd3a44ab7a46eeaf161b7e8a120"
server_key_version: "encryptionkey@0"
I20260926 20:10:43.259595 27428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "420020f3337c4b6e925f585db59b9993"
format_stamp: "Formatted at 2026-09-26 20:10:43 on dist-test-slave-7034"
server_key: "82cd78b8e521e43bb1ba42eb253c285b"
server_key_iv: "2a589dd3a44ab7a46eeaf161b7e8a120"
server_key_version: "encryptionkey@0"
I20260926 20:10:43.263455 27428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:10:43.266103 27443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:43.267474 27428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:43.267621 27428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "420020f3337c4b6e925f585db59b9993"
format_stamp: "Formatted at 2026-09-26 20:10:43 on dist-test-slave-7034"
server_key: "82cd78b8e521e43bb1ba42eb253c285b"
server_key_iv: "2a589dd3a44ab7a46eeaf161b7e8a120"
server_key_version: "encryptionkey@0"
I20260926 20:10:43.267745 27428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:43.286314 27428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:43.289949 27428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:43.290162 27428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:43.290867 27428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:43.291760 27428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:43.291819 27428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:43.291860 27428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:43.291916 27428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:43.300940 27428 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:44653
I20260926 20:10:43.301015 27557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:44653 every 8 connection(s)
I20260926 20:10:43.302021 27428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:10:43.305939 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 27428
I20260926 20:10:43.306077 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:10:43.306362 23754 external_mini_cluster.cc:1428] Setting key a8e75292cf0bce119b9068c10f160271
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453443, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:10:43.317003 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:43.303957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:39417 (local address 127.23.50.190:37403)
0926 20:10:43.304125 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:43.304130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:43.305095 (+ 965us) server_negotiation.cc:407] Connection header received
0926 20:10:43.306742 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:43.306746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:43.306805 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:43.306925 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:43.308185 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.308766 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.310085 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.310347 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.310474 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:43.313232 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:43.313235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:43.313237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:43.313259 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:43.314777 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:43.315322 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:43.315331 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:43.315332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:43.315405 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:43.315761 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:43.315765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:43.315767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:43.315957 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:43.316065 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:43.316690 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 20:10:43.316825 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
I20260926 20:10:43.317996 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:43.304364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37403 (local address 127.23.50.129:39417)
0926 20:10:43.304919 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:43.304980 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:43.306509 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:43.307110 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:43.307117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:43.307497 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:43.308000 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:43.308015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.308905 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.308909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:43.309953 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:43.309963 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.310513 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.310518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:43.310713 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:43.311387 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:43.311404 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:43.313043 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:43.314909 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:43.314915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:43.314930 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:43.315202 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:43.315507 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:43.315510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:43.315512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:43.315660 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:43.316077 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:43.316085 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:43.316418 (+ 333us) client_negotiation.cc:769] Sending connection context
0926 20:10:43.316646 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 20:10:43.317049 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":120,"threads_started":1}
I20260926 20:10:43.319252 27558 heartbeater.cc:347] Connected to a master server at 127.23.50.190:37403
I20260926 20:10:43.319520 27558 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:43.320067 27558 heartbeater.cc:511] Master 127.23.50.190:37403 requested a full tablet report, sending...
I20260926 20:10:43.321681 27366 ts_manager.cc:194] Registered new tserver with Master: 420020f3337c4b6e925f585db59b9993 (127.23.50.129:44653)
I20260926 20:10:43.322969 27366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:39417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:10:43.333636 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37403
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:43.445497 27563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:43.445765 27563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:43.445809 27563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:43.449981 27563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:43.450044 27563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:43.450106 27563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:10:43.455425 27563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:43.456507 27563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:43.457505 27563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:43.467489 27569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:43.467500 27571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:43.467500 27568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:43.467975 27563 server_base.cc:1034] running on GCE node
I20260926 20:10:43.468376 27563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:43.469060 27563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:43.470247 27563 hybrid_clock.cc:648] HybridClock initialized: now 1790453443470220 us; error 47 us; skew 500 ppm
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453443, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:43.473291 27563 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:10:43.474383 27563 webserver.cc:460] Webserver started at http://127.23.50.130:35857/ using document root <none> and password file <none>
I20260926 20:10:43.474961 27563 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:43.475052 27563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:43.475298 27563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:43.477363 27563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "4d9aa84eac144050bf095649080a21af"
format_stamp: "Formatted at 2026-09-26 20:10:43 on dist-test-slave-7034"
server_key: "3eb693eead5e292bb733a73eddcaffc0"
server_key_iv: "1b805ebe6d110405e8c74c186244d545"
server_key_version: "encryptionkey@0"
I20260926 20:10:43.477994 27563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "4d9aa84eac144050bf095649080a21af"
format_stamp: "Formatted at 2026-09-26 20:10:43 on dist-test-slave-7034"
server_key: "3eb693eead5e292bb733a73eddcaffc0"
server_key_iv: "1b805ebe6d110405e8c74c186244d545"
server_key_version: "encryptionkey@0"
I20260926 20:10:43.481771 27563 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:10:43.484562 27578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:43.485997 27563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:43.486148 27563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "4d9aa84eac144050bf095649080a21af"
format_stamp: "Formatted at 2026-09-26 20:10:43 on dist-test-slave-7034"
server_key: "3eb693eead5e292bb733a73eddcaffc0"
server_key_iv: "1b805ebe6d110405e8c74c186244d545"
server_key_version: "encryptionkey@0"
I20260926 20:10:43.486269 27563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:43.526173 27563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:43.529901 27563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:43.530148 27563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:43.530818 27563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:43.531709 27563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:43.531785 27563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:43.531864 27563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:43.531914 27563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:43.542387 27563 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:37933
I20260926 20:10:43.542408 27692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:37933 every 8 connection(s)
I20260926 20:10:43.543366 27563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:10:43.550931 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 27563
I20260926 20:10:43.551066 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:10:43.551324 23754 external_mini_cluster.cc:1428] Setting key 149cb9c4877403019d198d14f7e0d5ea
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453443, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:43.558586 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:43.545412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:58495 (local address 127.23.50.190:37403)
0926 20:10:43.545603 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:43.545608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:43.546553 (+ 945us) server_negotiation.cc:407] Connection header received
0926 20:10:43.547882 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:43.547884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:43.547931 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:43.548008 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:43.549324 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.549930 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.551105 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.551342 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.551457 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:43.554389 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:43.554394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:43.554400 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:43.554437 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:43.555917 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:43.556590 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:43.556593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:43.556594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:43.556676 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:43.557112 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:43.557114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:43.557115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:43.557353 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:43.557437 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:43.558190 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 20:10:43.558338 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 20:10:43.559193 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:43.545839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37403 (local address 127.23.50.130:58495)
0926 20:10:43.546397 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:43.546442 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:43.547660 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:43.548295 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:43.548303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:43.548675 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:43.549160 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:43.549173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.550042 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.550045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:43.550973 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:43.550982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.551500 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.551504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:43.551674 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:43.552325 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:43.552341 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:43.554202 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:43.556041 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:43.556050 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:43.556070 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:43.556465 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:43.556782 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:43.556788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:43.556793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:43.557003 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:43.557472 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:43.557479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:43.557786 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 20:10:43.558011 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 20:10:43.558294 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":114,"threads_started":1}
I20260926 20:10:43.560470 27693 heartbeater.cc:347] Connected to a master server at 127.23.50.190:37403
I20260926 20:10:43.560752 27693 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:43.561399 27693 heartbeater.cc:511] Master 127.23.50.190:37403 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:10:43.562606 27366 ts_manager.cc:194] Registered new tserver with Master: 4d9aa84eac144050bf095649080a21af (127.23.50.130:37933)
I20260926 20:10:43.563258 27366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:58495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:10:43.581622 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37403
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:43.694589 27698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:43.694859 27698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:43.694902 27698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:43.698942 27698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:43.698998 27698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:43.699056 27698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:10:43.704245 27698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:43.705256 27698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:43.706255 27698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:43.715996 27704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:43.716008 27706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:43.716187 27698 server_base.cc:1034] running on GCE node
W20260926 20:10:43.716013 27703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:43.716710 27698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:43.717392 27698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:43.718585 27698 hybrid_clock.cc:648] HybridClock initialized: now 1790453443718548 us; error 50 us; skew 500 ppm
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453443, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:43.721473 27698 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:10:43.722524 27698 webserver.cc:460] Webserver started at http://127.23.50.131:38003/ using document root <none> and password file <none>
I20260926 20:10:43.723114 27698 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:43.723205 27698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:43.723428 27698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:43.725476 27698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "99314f05666f4c96ad8d590ffb2d967a"
format_stamp: "Formatted at 2026-09-26 20:10:43 on dist-test-slave-7034"
server_key: "f620894eab4779bcf6718d7a90d0bb9c"
server_key_iv: "6e256299a0005f625afd6a159307cc09"
server_key_version: "encryptionkey@0"
I20260926 20:10:43.726094 27698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "99314f05666f4c96ad8d590ffb2d967a"
format_stamp: "Formatted at 2026-09-26 20:10:43 on dist-test-slave-7034"
server_key: "f620894eab4779bcf6718d7a90d0bb9c"
server_key_iv: "6e256299a0005f625afd6a159307cc09"
server_key_version: "encryptionkey@0"
I20260926 20:10:43.730182 27698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:10:43.732807 27713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:43.734414 27698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:43.734568 27698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "99314f05666f4c96ad8d590ffb2d967a"
format_stamp: "Formatted at 2026-09-26 20:10:43 on dist-test-slave-7034"
server_key: "f620894eab4779bcf6718d7a90d0bb9c"
server_key_iv: "6e256299a0005f625afd6a159307cc09"
server_key_version: "encryptionkey@0"
I20260926 20:10:43.734699 27698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:43.753409 27698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:43.756844 27698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:43.757066 27698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:43.757700 27698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:43.758672 27698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:43.758754 27698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:43.758839 27698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:43.758898 27698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:43.767938 27698 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:42533
I20260926 20:10:43.767961 27826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:42533 every 8 connection(s)
I20260926 20:10:43.768896 27698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:10:43.778410 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 27698
I20260926 20:10:43.778555 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453443, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:43.778889 23754 external_mini_cluster.cc:1428] Setting key dc0aa364816d5396dc5ba750bafa91b6
I20260926 20:10:43.783160 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:43.770829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:33341 (local address 127.23.50.190:37403)
0926 20:10:43.770982 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:43.770987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:43.771862 (+ 875us) server_negotiation.cc:407] Connection header received
0926 20:10:43.773248 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:43.773250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:43.773300 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:43.773416 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:43.774717 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.775271 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.776422 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.776659 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.776770 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:43.779456 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:43.779459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:43.779461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:43.779486 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:43.781291 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:43.781734 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:43.781737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:43.781738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:43.781812 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:43.782163 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:43.782165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:43.782166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:43.782365 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:43.782443 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:43.782951 (+ 508us) server_negotiation.cc:299] Negotiation successful
0926 20:10:43.783048 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:10:43.784004 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:43.771222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37403 (local address 127.23.50.131:33341)
0926 20:10:43.771668 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:43.771709 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:43.773045 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:43.773596 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:43.773603 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:43.774030 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:43.774572 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:43.774587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.775400 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.775405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:43.776295 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:43.776305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.776812 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.776816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:43.776987 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:43.777538 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:43.777553 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:43.779275 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:43.781387 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:43.781391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:43.781403 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:43.781652 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:43.781945 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:43.781947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:43.781949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:43.782082 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:43.782494 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:43.782499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:43.782742 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 20:10:43.782910 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 20:10:43.783160 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":106,"threads_started":1}
I20260926 20:10:43.785147 27827 heartbeater.cc:347] Connected to a master server at 127.23.50.190:37403
I20260926 20:10:43.785403 27827 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:43.785948 27827 heartbeater.cc:511] Master 127.23.50.190:37403 requested a full tablet report, sending...
I20260926 20:10:43.786984 27366 ts_manager.cc:194] Registered new tserver with Master: 99314f05666f4c96ad8d590ffb2d967a (127.23.50.131:42533)
I20260926 20:10:43.787513 27366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:33341
I20260926 20:10:43.792855 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.129@KRBTEST.COM
I20260926 20:10:43.818440 27834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:43.809293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.23.50.129:44653)
0926 20:10:43.809556 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:43.809575 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:43.809608 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:10:43.809661 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:43.809666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:43.809920 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:43.810058 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:43.810690 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.811488 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.812285 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.812656 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.812743 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:43.815113 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:43.815123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:43.815137 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:43.815201 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:43.816747 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:43.817143 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:43.817149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:43.817150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:43.817227 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:43.817483 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:43.817486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:43.817488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:43.817752 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:43.817997 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:43.818135 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 20:10:43.818287 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:43.829668 27405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:43.821453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.23.50.190:37403)
0926 20:10:43.821591 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:43.821597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:43.821659 (+ 62us) server_negotiation.cc:407] Connection header received
0926 20:10:43.821836 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:43.821839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:43.821933 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:43.822016 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:43.822632 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.823146 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.823957 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:43.824182 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:43.824292 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:43.826663 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:43.826666 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:43.826669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:43.826697 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:43.827960 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:43.828424 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:43.828428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:43.828432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:43.828512 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:43.828788 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:43.828792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:43.828794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:43.829013 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:43.829156 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:43.829382 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 20:10:43.829502 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 20:10:43.830096 27366 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39418
I20260926 20:10:43.830694 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 27428
I20260926 20:10:43.836045 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 27563
I20260926 20:10:43.841339 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 27698
I20260926 20:10:43.846243 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 27334
2026-09-26T20:10:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2194 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27839](info): set up 2 sockets
Sep 26 20:10:43 dist-test-slave-7034 krb5kdc[27839](info): commencing operation
krb5kdc: starting...
W20260926 20:10:44.902536 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:10:44 dist-test-slave-7034 krb5kdc[27839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:10:45.033311 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42899
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:35077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:42899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:45.152225 27852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:45.152499 27852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:45.152542 27852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:45.156582 27852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:45.156647 27852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:45.156666 27852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:45.156682 27852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:45.156697 27852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:45.162110 27852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:35077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:42899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:45.163169 27852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:45.164147 27852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:45.172730 27858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:45.172895 27852 server_base.cc:1034] running on GCE node
W20260926 20:10:45.172640 27860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:45.172730 27857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:45.173624 27852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:45.174855 27852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:45.176062 27852 hybrid_clock.cc:648] HybridClock initialized: now 1790453445176038 us; error 36 us; skew 500 ppm
Sep 26 20:10:45 dist-test-slave-7034 krb5kdc[27839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453445, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:45.184085 27852 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:10:45.184918 27852 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:45.185022 27852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:45.185269 27852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:45.187497 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "b637dcfb729b4b1e89e9ed1f89499796"
format_stamp: "Formatted at 2026-09-26 20:10:45 on dist-test-slave-7034"
server_key: "ce4ddf4404de06d04d2a1c26b572f17b"
server_key_iv: "5f0c4c30b04cd27511e82390a925a9b0"
server_key_version: "encryptionkey@0"
I20260926 20:10:45.188088 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "b637dcfb729b4b1e89e9ed1f89499796"
format_stamp: "Formatted at 2026-09-26 20:10:45 on dist-test-slave-7034"
server_key: "ce4ddf4404de06d04d2a1c26b572f17b"
server_key_iv: "5f0c4c30b04cd27511e82390a925a9b0"
server_key_version: "encryptionkey@0"
I20260926 20:10:45.192037 27852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 20:10:45.194573 27866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:45.195883 27852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:45.196048 27852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "b637dcfb729b4b1e89e9ed1f89499796"
format_stamp: "Formatted at 2026-09-26 20:10:45 on dist-test-slave-7034"
server_key: "ce4ddf4404de06d04d2a1c26b572f17b"
server_key_iv: "5f0c4c30b04cd27511e82390a925a9b0"
server_key_version: "encryptionkey@0"
I20260926 20:10:45.196218 27852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:45.217518 27852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:45.221128 27852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:45.221395 27852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:45.229148 27852 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:42899
I20260926 20:10:45.229482 27918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:42899 every 8 connection(s)
I20260926 20:10:45.229980 27852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:45.230222 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 27852
I20260926 20:10:45.230355 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:45.230708 23754 external_mini_cluster.cc:1428] Setting key e467f56e2ef42cfa6700360c9f58db51
I20260926 20:10:45.233327 27919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:45.238968 27919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796: Bootstrap starting.
Sep 26 20:10:45 dist-test-slave-7034 krb5kdc[27839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:45.241572 27919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:45.242624 27919 log.cc:824] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:45.245239 27919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796: No bootstrap required, opened a new log
I20260926 20:10:45.246598 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:45.232160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.23.50.190:42899)
0926 20:10:45.232638 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:45.232655 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:45.232693 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:10:45.233472 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:45.233500 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:45.233928 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:45.234291 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:45.235172 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.236255 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.237196 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.237714 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.237963 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:45.241037 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:45.241043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:45.241058 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:45.241089 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:45.243304 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:45.243888 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:45.243895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:45.243901 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:45.244037 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:45.244328 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:45.244331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:45.244333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:45.244856 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:45.245049 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:45.245314 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 20:10:45.245625 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":141,"threads_started":1}
I20260926 20:10:45.249590 27919 raft_consensus.cc:348] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b637dcfb729b4b1e89e9ed1f89499796" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42899 } }
I20260926 20:10:45.249827 27919 raft_consensus.cc:374] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:45.249953 27919 raft_consensus.cc:729] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b637dcfb729b4b1e89e9ed1f89499796, State: Initialized, Role: FOLLOWER
I20260926 20:10:45.250375 27919 consensus_queue.cc:260] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b637dcfb729b4b1e89e9ed1f89499796" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42899 } }
I20260926 20:10:45.250499 27919 raft_consensus.cc:388] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:45.250538 27919 raft_consensus.cc:482] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:45.250602 27919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:45.251497 27919 raft_consensus.cc:504] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b637dcfb729b4b1e89e9ed1f89499796" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42899 } }
I20260926 20:10:45.251802 27919 leader_election.cc:302] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b637dcfb729b4b1e89e9ed1f89499796; no voters:
I20260926 20:10:45.252087 27919 leader_election.cc:288] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:45.252345 27924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:45.252655 27924 raft_consensus.cc:686] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [term 1 LEADER]: Becoming Leader. State: Replica: b637dcfb729b4b1e89e9ed1f89499796, State: Running, Role: LEADER
I20260926 20:10:45.253033 27924 consensus_queue.cc:237] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b637dcfb729b4b1e89e9ed1f89499796" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42899 } }
I20260926 20:10:45.253414 27919 sys_catalog.cc:564] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:45.254725 27926 sys_catalog.cc:455] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b637dcfb729b4b1e89e9ed1f89499796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b637dcfb729b4b1e89e9ed1f89499796" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42899 } } }
I20260926 20:10:45.254868 27926 sys_catalog.cc:458] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:45.255183 27927 sys_catalog.cc:455] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b637dcfb729b4b1e89e9ed1f89499796. Latest consensus state: current_term: 1 leader_uuid: "b637dcfb729b4b1e89e9ed1f89499796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b637dcfb729b4b1e89e9ed1f89499796" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42899 } } }
I20260926 20:10:45.255275 27927 sys_catalog.cc:458] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:45.255362 27934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:45.258608 27934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:45.264613 27934 catalog_manager.cc:1347] Generated new cluster ID: 5695031326ad4f528f319665f5d4958f
I20260926 20:10:45.264688 27934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:45.293084 27934 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:45.294530 27934 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:45.305001 27934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796: Generated new TSK 0
I20260926 20:10:45.305969 27934 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:10:45.338038 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42899
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:35077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:45.454042 27947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:45.454383 27947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:45.454457 27947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:45.458703 27947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:45.458760 27947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:45.458847 27947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:10:45.464100 27947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:35077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:45.465194 27947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:45.466277 27947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:10:45.476531 27947 server_base.cc:1034] running on GCE node
W20260926 20:10:45.476490 27953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:45.476460 27955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:45.476460 27952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:45.477310 27947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:45.478022 27947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:45.479252 27947 hybrid_clock.cc:648] HybridClock initialized: now 1790453445479224 us; error 37 us; skew 500 ppm
Sep 26 20:10:45 dist-test-slave-7034 krb5kdc[27839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453445, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:45.482101 27947 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:10:45.482874 27947 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:45.482971 27947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:45.483219 27947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:45.485288 27947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "faa8854d1d214547b99cf778591b65c1"
format_stamp: "Formatted at 2026-09-26 20:10:45 on dist-test-slave-7034"
server_key: "61ce26d784bf6a8f95a7fb9f1deb6839"
server_key_iv: "80a07db6f74d9a0816c3e823c32270fe"
server_key_version: "encryptionkey@0"
I20260926 20:10:45.485912 27947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "faa8854d1d214547b99cf778591b65c1"
format_stamp: "Formatted at 2026-09-26 20:10:45 on dist-test-slave-7034"
server_key: "61ce26d784bf6a8f95a7fb9f1deb6839"
server_key_iv: "80a07db6f74d9a0816c3e823c32270fe"
server_key_version: "encryptionkey@0"
I20260926 20:10:45.489642 27947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:10:45.492193 27961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:45.493467 27947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:45.493620 27947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "faa8854d1d214547b99cf778591b65c1"
format_stamp: "Formatted at 2026-09-26 20:10:45 on dist-test-slave-7034"
server_key: "61ce26d784bf6a8f95a7fb9f1deb6839"
server_key_iv: "80a07db6f74d9a0816c3e823c32270fe"
server_key_version: "encryptionkey@0"
I20260926 20:10:45.493737 27947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:45.512043 27947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:45.515405 27947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:45.515602 27947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:45.516173 27947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:45.517053 27947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:45.517113 27947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:45.517174 27947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:45.517199 27947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:45.526186 27947 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:34087
I20260926 20:10:45.526202 28074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:34087 every 8 connection(s)
I20260926 20:10:45.526778 27947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:10:45.535264 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 27947
I20260926 20:10:45.535369 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:10:45.535647 23754 external_mini_cluster.cc:1428] Setting key 4be40cfdae9540a5bf8dd1b537c14213
Sep 26 20:10:45 dist-test-slave-7034 krb5kdc[27839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453445, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:45.541481 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:45.528536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:39201 (local address 127.23.50.190:42899)
0926 20:10:45.528673 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:45.528676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:45.529480 (+ 804us) server_negotiation.cc:407] Connection header received
0926 20:10:45.530724 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:45.530727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:45.530789 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:45.530864 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:45.531979 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.532518 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.533633 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.533961 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.534046 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:45.536584 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:45.536588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:45.536590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:45.536612 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:45.539166 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:45.539690 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:45.539697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:45.539700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:45.539802 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:45.540155 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:45.540160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:45.540165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:45.540437 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:45.540544 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:45.541103 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 20:10:45.541261 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:10:45.542366 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:45.528861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42899 (local address 127.23.50.129:39201)
0926 20:10:45.529304 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:45.529347 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:45.530526 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:45.531026 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:45.531033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:45.531376 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:45.531845 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:45.531858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.532629 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.532632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:45.533515 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:45.533525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.534100 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.534104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:45.534250 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:45.534840 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:45.534857 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:45.536436 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:45.539307 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:45.539312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:45.539324 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:45.539582 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:45.539903 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:45.539905 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:45.539907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:45.540036 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:45.540577 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:45.540582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:45.540835 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 20:10:45.541007 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 20:10:45.541396 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
I20260926 20:10:45.543782 28075 heartbeater.cc:347] Connected to a master server at 127.23.50.190:42899
I20260926 20:10:45.544059 28075 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:45.544634 28075 heartbeater.cc:511] Master 127.23.50.190:42899 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:10:45.546445 27883 ts_manager.cc:194] Registered new tserver with Master: faa8854d1d214547b99cf778591b65c1 (127.23.50.129:34087)
I20260926 20:10:45.547801 27883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:39201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:10:45.566169 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42899
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:35077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:45.678936 28080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:45.679242 28080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:45.679339 28080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:45.683442 28080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:45.683530 28080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:45.683620 28080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:10:45.689013 28080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:35077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:45.690258 28080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:45.691305 28080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:45.700922 28086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:45.700924 28088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:45.701146 28080 server_base.cc:1034] running on GCE node
W20260926 20:10:45.700924 28085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:45.701601 28080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:45.702297 28080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:45.703491 28080 hybrid_clock.cc:648] HybridClock initialized: now 1790453445703475 us; error 30 us; skew 500 ppm
Sep 26 20:10:45 dist-test-slave-7034 krb5kdc[27839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453445, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:45.706190 28080 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:10:45.706933 28080 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:45.706997 28080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:45.707158 28080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:45.709163 28080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "b723050cd3ee495cb965a2bb373f8bf2"
format_stamp: "Formatted at 2026-09-26 20:10:45 on dist-test-slave-7034"
server_key: "9f1e7c7e4f70621e0544429783251add"
server_key_iv: "5dae662cc400bdc1057926dfec621474"
server_key_version: "encryptionkey@0"
I20260926 20:10:45.709679 28080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "b723050cd3ee495cb965a2bb373f8bf2"
format_stamp: "Formatted at 2026-09-26 20:10:45 on dist-test-slave-7034"
server_key: "9f1e7c7e4f70621e0544429783251add"
server_key_iv: "5dae662cc400bdc1057926dfec621474"
server_key_version: "encryptionkey@0"
I20260926 20:10:45.713248 28080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:10:45.715652 28094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:45.716948 28080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 20:10:45.717085 28080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "b723050cd3ee495cb965a2bb373f8bf2"
format_stamp: "Formatted at 2026-09-26 20:10:45 on dist-test-slave-7034"
server_key: "9f1e7c7e4f70621e0544429783251add"
server_key_iv: "5dae662cc400bdc1057926dfec621474"
server_key_version: "encryptionkey@0"
I20260926 20:10:45.717202 28080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:45.728746 28080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:45.732115 28080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:45.732332 28080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:45.732918 28080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:45.733769 28080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:45.733843 28080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:45.733989 28080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:45.734043 28080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:45.743287 28080 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:43535
I20260926 20:10:45.743295 28207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:43535 every 8 connection(s)
I20260926 20:10:45.743937 28080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:10:45.752769 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 28080
I20260926 20:10:45.752914 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:10:45.753232 23754 external_mini_cluster.cc:1428] Setting key b5345654655a48342f6e68bda90f30f7
Sep 26 20:10:45 dist-test-slave-7034 krb5kdc[27839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453445, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:45.758782 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:45.745834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:42407 (local address 127.23.50.190:42899)
0926 20:10:45.746036 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:45.746041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:45.746950 (+ 909us) server_negotiation.cc:407] Connection header received
0926 20:10:45.748306 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:45.748308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:45.748369 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:45.748452 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:45.749720 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.750378 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.751542 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.751760 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.751850 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:45.754871 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:45.754874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:45.754877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:45.754902 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:45.756548 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:45.757068 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:45.757075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:45.757078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:45.757189 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:45.757530 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:45.757536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:45.757538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:45.757801 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:45.757965 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:45.758483 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 20:10:45.758611 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:10:45.759756 28210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:45.746312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42899 (local address 127.23.50.130:42407)
0926 20:10:45.746788 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:45.746834 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:45.748046 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:45.748660 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:45.748670 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:45.749045 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:45.749572 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:45.749586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.750496 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.750499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:45.751419 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:45.751429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.751899 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.751902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:45.752039 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:45.752671 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:45.752694 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:45.754670 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:45.756678 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:45.756683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:45.756701 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:45.756965 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:45.757287 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:45.757290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:45.757291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:45.757430 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:45.757968 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:45.757973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:45.758234 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 20:10:45.758438 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 20:10:45.758806 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260926 20:10:45.761241 28208 heartbeater.cc:347] Connected to a master server at 127.23.50.190:42899
I20260926 20:10:45.761514 28208 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:45.762049 28208 heartbeater.cc:511] Master 127.23.50.190:42899 requested a full tablet report, sending...
I20260926 20:10:45.763298 27883 ts_manager.cc:194] Registered new tserver with Master: b723050cd3ee495cb965a2bb373f8bf2 (127.23.50.130:43535)
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:10:45.764143 27883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:42407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:10:45.785372 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42899
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:35077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:45.898115 28213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:45.898375 28213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:45.898455 28213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:45.902657 28213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:45.902714 28213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:45.902802 28213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:10:45.908068 28213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:35077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:45.909116 28213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:45.910153 28213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:10:45.919839 28213 server_base.cc:1034] running on GCE node
W20260926 20:10:45.919771 28221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:45.919800 28218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:45.919812 28219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:45.920677 28213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:45.921483 28213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:45.922683 28213 hybrid_clock.cc:648] HybridClock initialized: now 1790453445922655 us; error 47 us; skew 500 ppm
Sep 26 20:10:45 dist-test-slave-7034 krb5kdc[27839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453445, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:45.925544 28213 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:10:45.926393 28213 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:45.926491 28213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:45.926728 28213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:45.928810 28213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "2fff3239a8cf4483b50f7a10f07b4de0"
format_stamp: "Formatted at 2026-09-26 20:10:45 on dist-test-slave-7034"
server_key: "21e2e1c3cdf0a7041fb8d0dd3a6cc7b5"
server_key_iv: "37e6422775e461e2f0035f4c8e7d7bf2"
server_key_version: "encryptionkey@0"
I20260926 20:10:45.929409 28213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "2fff3239a8cf4483b50f7a10f07b4de0"
format_stamp: "Formatted at 2026-09-26 20:10:45 on dist-test-slave-7034"
server_key: "21e2e1c3cdf0a7041fb8d0dd3a6cc7b5"
server_key_iv: "37e6422775e461e2f0035f4c8e7d7bf2"
server_key_version: "encryptionkey@0"
I20260926 20:10:45.933416 28213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:10:45.935971 28227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:45.937234 28213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:45.937386 28213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "2fff3239a8cf4483b50f7a10f07b4de0"
format_stamp: "Formatted at 2026-09-26 20:10:45 on dist-test-slave-7034"
server_key: "21e2e1c3cdf0a7041fb8d0dd3a6cc7b5"
server_key_iv: "37e6422775e461e2f0035f4c8e7d7bf2"
server_key_version: "encryptionkey@0"
I20260926 20:10:45.937503 28213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:45.948668 28213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:45.952086 28213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:45.952291 28213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:45.952880 28213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:45.953919 28213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:45.953986 28213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:45.954059 28213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:45.954102 28213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:45.963387 28213 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:41841
I20260926 20:10:45.963438 28340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:41841 every 8 connection(s)
I20260926 20:10:45.964030 28213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:10:45.972034 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 28213
I20260926 20:10:45.972193 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:10:45.972491 23754 external_mini_cluster.cc:1428] Setting key 0bc8cbe9e7da8d2e3592faf71046ed9f
Sep 26 20:10:45 dist-test-slave-7034 krb5kdc[27839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453445, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:45.979029 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:45.966035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:58143 (local address 127.23.50.190:42899)
0926 20:10:45.966216 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:45.966221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:45.967171 (+ 950us) server_negotiation.cc:407] Connection header received
0926 20:10:45.968504 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:45.968507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:45.968556 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:45.968634 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:45.969927 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.970486 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.971689 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.972043 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.972176 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:45.975348 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:45.975351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:45.975353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:45.975381 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:45.976722 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:45.977358 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:45.977361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:45.977363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:45.977455 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:45.977835 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:45.977837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:45.977838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:45.978085 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:45.978166 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:45.978793 (+ 627us) server_negotiation.cc:299] Negotiation successful
0926 20:10:45.978906 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 20:10:45.979920 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:45.966483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42899 (local address 127.23.50.131:58143)
0926 20:10:45.967011 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:45.967056 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:45.968300 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:45.968861 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:45.968868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:45.969243 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:45.969741 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:45.969755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.970607 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.970609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:45.971563 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:45.971573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.972347 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.972351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:45.972511 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:45.973192 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:45.973215 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:45.975159 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:45.976866 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:45.976871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:45.976886 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:45.977252 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:45.977575 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:45.977578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:45.977579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:45.977744 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:45.978209 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:45.978215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:45.978533 (+ 318us) client_negotiation.cc:769] Sending connection context
0926 20:10:45.978713 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 20:10:45.979010 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":125,"threads_started":1}
I20260926 20:10:45.981168 28341 heartbeater.cc:347] Connected to a master server at 127.23.50.190:42899
I20260926 20:10:45.981462 28341 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:45.981993 28341 heartbeater.cc:511] Master 127.23.50.190:42899 requested a full tablet report, sending...
I20260926 20:10:45.983103 27883 ts_manager.cc:194] Registered new tserver with Master: 2fff3239a8cf4483b50f7a10f07b4de0 (127.23.50.131:41841)
I20260926 20:10:45.983701 27883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:58143
I20260926 20:10:45.986846 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:10:45.996506 27922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:45.988788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.23.50.190:42899)
0926 20:10:45.988910 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:45.988913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:45.989135 (+ 222us) server_negotiation.cc:407] Connection header received
0926 20:10:45.989244 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:45.989246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:45.989309 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:45.989378 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:45.990244 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.990754 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.991665 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:45.991874 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:45.991951 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:45.993329 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:45.993331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:45.993334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:45.993356 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:45.994823 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:45.995204 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:45.995207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:45.995208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:45.995302 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:45.995534 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:45.995536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:45.995537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:45.995748 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:45.995853 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:45.996200 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 20:10:45.996383 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:10:45.999678 27883 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:10:46.002956 27883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:46.017997 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.011861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:41841 (local address 127.0.0.1:59890)
0926 20:10:46.012835 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:46.012849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:46.012928 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:46.013541 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:46.013546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:46.013582 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:46.013782 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:46.013788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.015060 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.015064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:46.016282 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:46.016291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.017412 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.017416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:46.017522 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:46.017557 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:10:46.017673 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 20:10:46.017797 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":90,"spinlock_wait_cycles":6784,"thread_start_us":94,"threads_started":1}
I20260926 20:10:46.018023 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.012583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.23.50.131:41841)
0926 20:10:46.013051 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:46.013062 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:46.013083 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:10:46.013156 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:46.013160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:46.013354 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:46.013547 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:46.013933 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.014913 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.016406 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.017278 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.017386 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:46.017540 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:46.017719 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:10:46.017829 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":159,"threads_started":1}
I20260926 20:10:46.018329 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.011911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.23.50.129:34087)
0926 20:10:46.012839 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:46.012848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:46.012875 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:10:46.012954 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:46.012963 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:46.013189 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:46.013344 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:46.014008 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.014944 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.016675 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.017315 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.017440 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:46.017579 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:46.018062 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 20:10:46.018170 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":84,"threads_started":1}
I20260926 20:10:46.017997 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.011799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:34087 (local address 127.0.0.1:40600)
0926 20:10:46.012379 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:46.012430 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:46.012552 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:46.013540 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:46.013548 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:46.013583 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:46.013828 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:46.013836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.015096 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.015099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:46.016536 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:46.016547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.017641 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.017648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:46.017731 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:46.017750 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:10:46.017825 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:10:46.017902 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":128,"spinlock_wait_cycles":2048,"thread_start_us":103,"threads_started":1}
I20260926 20:10:46.019132 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.012510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:43535 (local address 127.0.0.1:58008)
0926 20:10:46.013318 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:46.013330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:46.013463 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:46.014198 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:46.014201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:46.014219 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:46.014327 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:46.014333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.015681 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.015684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:46.016837 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:46.016846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.018681 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.018685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:46.018786 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:46.018809 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:10:46.018894 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:10:46.018985 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":34,"thread_start_us":107,"threads_started":1}
I20260926 20:10:46.019199 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.012775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.23.50.130:43535)
0926 20:10:46.013166 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:46.013177 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:46.013399 (+ 222us) server_negotiation.cc:407] Connection header received
0926 20:10:46.013626 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:46.013634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:46.013871 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:46.013987 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:46.014457 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.015524 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.016964 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.017774 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.017908 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:46.018041 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:46.018939 (+ 898us) server_negotiation.cc:299] Negotiation successful
0926 20:10:46.019052 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":106,"threads_started":1}
I20260926 20:10:46.021209 28275 tablet_service.cc:1467] Processing CreateTablet for tablet 5796c9e4af4b49d6b6358dd91aefc8c7 (DEFAULT_TABLE table=test-table [id=b129675a837e4cb3ad4e945eff6d6cf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:46.021690 28009 tablet_service.cc:1467] Processing CreateTablet for tablet 5796c9e4af4b49d6b6358dd91aefc8c7 (DEFAULT_TABLE table=test-table [id=b129675a837e4cb3ad4e945eff6d6cf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:46.022360 28275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5796c9e4af4b49d6b6358dd91aefc8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:46.022105 28142 tablet_service.cc:1467] Processing CreateTablet for tablet 5796c9e4af4b49d6b6358dd91aefc8c7 (DEFAULT_TABLE table=test-table [id=b129675a837e4cb3ad4e945eff6d6cf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:46.022747 28009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5796c9e4af4b49d6b6358dd91aefc8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:46.023039 28142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5796c9e4af4b49d6b6358dd91aefc8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:46.028379 28358 tablet_bootstrap.cc:492] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0: Bootstrap starting.
I20260926 20:10:46.030614 28359 tablet_bootstrap.cc:492] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2: Bootstrap starting.
I20260926 20:10:46.030778 28360 tablet_bootstrap.cc:492] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1: Bootstrap starting.
I20260926 20:10:46.031136 28358 tablet_bootstrap.cc:654] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:46.032215 28358 log.cc:824] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:46.032819 28360 tablet_bootstrap.cc:654] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:46.032819 28359 tablet_bootstrap.cc:654] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:46.034171 28358 tablet_bootstrap.cc:492] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0: No bootstrap required, opened a new log
I20260926 20:10:46.034158 28359 log.cc:824] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:46.034169 28360 log.cc:824] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:46.034456 28358 ts_tablet_manager.cc:1397] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:10:46.036094 28360 tablet_bootstrap.cc:492] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1: No bootstrap required, opened a new log
I20260926 20:10:46.036094 28359 tablet_bootstrap.cc:492] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2: No bootstrap required, opened a new log
I20260926 20:10:46.036280 28360 ts_tablet_manager.cc:1397] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:10:46.036280 28359 ts_tablet_manager.cc:1397] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:10:46.037588 28358 raft_consensus.cc:348] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa8854d1d214547b99cf778591b65c1" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 34087 } } peers { permanent_uuid: "2fff3239a8cf4483b50f7a10f07b4de0" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41841 } } peers { permanent_uuid: "b723050cd3ee495cb965a2bb373f8bf2" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43535 } }
I20260926 20:10:46.037848 28358 raft_consensus.cc:374] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:46.037930 28358 raft_consensus.cc:729] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fff3239a8cf4483b50f7a10f07b4de0, State: Initialized, Role: FOLLOWER
I20260926 20:10:46.038381 28358 consensus_queue.cc:260] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa8854d1d214547b99cf778591b65c1" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 34087 } } peers { permanent_uuid: "2fff3239a8cf4483b50f7a10f07b4de0" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41841 } } peers { permanent_uuid: "b723050cd3ee495cb965a2bb373f8bf2" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43535 } }
I20260926 20:10:46.039059 28341 heartbeater.cc:503] Master 127.23.50.190:42899 was elected leader, sending a full tablet report...
I20260926 20:10:46.039381 28358 ts_tablet_manager.cc:1428] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260926 20:10:46.040066 28359 raft_consensus.cc:348] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa8854d1d214547b99cf778591b65c1" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 34087 } } peers { permanent_uuid: "2fff3239a8cf4483b50f7a10f07b4de0" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41841 } } peers { permanent_uuid: "b723050cd3ee495cb965a2bb373f8bf2" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43535 } }
I20260926 20:10:46.040341 28359 raft_consensus.cc:374] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:46.040396 28359 raft_consensus.cc:729] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b723050cd3ee495cb965a2bb373f8bf2, State: Initialized, Role: FOLLOWER
I20260926 20:10:46.040845 28359 consensus_queue.cc:260] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa8854d1d214547b99cf778591b65c1" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 34087 } } peers { permanent_uuid: "2fff3239a8cf4483b50f7a10f07b4de0" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41841 } } peers { permanent_uuid: "b723050cd3ee495cb965a2bb373f8bf2" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43535 } }
I20260926 20:10:46.040786 28360 raft_consensus.cc:348] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa8854d1d214547b99cf778591b65c1" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 34087 } } peers { permanent_uuid: "2fff3239a8cf4483b50f7a10f07b4de0" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41841 } } peers { permanent_uuid: "b723050cd3ee495cb965a2bb373f8bf2" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43535 } }
I20260926 20:10:46.041014 28360 raft_consensus.cc:374] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:46.041070 28360 raft_consensus.cc:729] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa8854d1d214547b99cf778591b65c1, State: Initialized, Role: FOLLOWER
I20260926 20:10:46.041501 28360 consensus_queue.cc:260] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa8854d1d214547b99cf778591b65c1" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 34087 } } peers { permanent_uuid: "2fff3239a8cf4483b50f7a10f07b4de0" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41841 } } peers { permanent_uuid: "b723050cd3ee495cb965a2bb373f8bf2" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43535 } }
I20260926 20:10:46.041903 28359 ts_tablet_manager.cc:1428] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:10:46.042181 28075 heartbeater.cc:503] Master 127.23.50.190:42899 was elected leader, sending a full tablet report...
I20260926 20:10:46.042225 28208 heartbeater.cc:503] Master 127.23.50.190:42899 was elected leader, sending a full tablet report...
I20260926 20:10:46.043474 28360 ts_tablet_manager.cc:1428] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 20:10:46.052825 28366 raft_consensus.cc:482] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:46.053043 28366 raft_consensus.cc:504] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa8854d1d214547b99cf778591b65c1" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 34087 } } peers { permanent_uuid: "2fff3239a8cf4483b50f7a10f07b4de0" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41841 } } peers { permanent_uuid: "b723050cd3ee495cb965a2bb373f8bf2" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43535 } }
I20260926 20:10:46.054327 28366 leader_election.cc:288] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fff3239a8cf4483b50f7a10f07b4de0 (127.23.50.131:41841), b723050cd3ee495cb965a2bb373f8bf2 (127.23.50.130:43535)
I20260926 20:10:46.058612 28356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.054738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:33067 (local address 127.23.50.131:41841)
0926 20:10:46.054894 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:46.054899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:46.055029 (+ 130us) server_negotiation.cc:407] Connection header received
0926 20:10:46.055215 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:46.055219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:46.055267 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:46.055342 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:46.055752 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.056461 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.057530 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.058034 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.058157 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:46.058211 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:46.058413 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 20:10:46.058477 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 20:10:46.058624 28367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.054563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:41841 (local address 127.23.50.129:33067)
0926 20:10:46.054922 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:46.054938 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:46.055052 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:46.055475 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:46.055481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:46.055500 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:46.055629 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:46.055636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.056571 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.056575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:46.057399 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:46.057409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.058151 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.058154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:46.058254 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:46.058289 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:10:46.058393 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:10:46.058474 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":93,"threads_started":1}
I20260926 20:10:46.058717 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.054824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:56335 (local address 127.23.50.130:43535)
0926 20:10:46.054938 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:46.054943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:46.055238 (+ 295us) server_negotiation.cc:407] Connection header received
0926 20:10:46.055358 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:46.055362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:46.055412 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:46.055507 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:46.055843 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.056617 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.057649 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.058160 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.058295 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:46.058348 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:46.058528 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 20:10:46.058599 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:10:46.058740 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.054706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:43535 (local address 127.23.50.129:56335)
0926 20:10:46.055129 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:46.055143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:46.055250 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:46.055524 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:46.055527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:46.055545 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:46.055659 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:46.055664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.056743 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.056749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:46.057523 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:46.057532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.058280 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.058284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:46.058397 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:46.058423 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:10:46.058513 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:10:46.058584 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":96,"threads_started":1}
I20260926 20:10:46.059299 28295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5796c9e4af4b49d6b6358dd91aefc8c7" candidate_uuid: "faa8854d1d214547b99cf778591b65c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fff3239a8cf4483b50f7a10f07b4de0" is_pre_election: true
I20260926 20:10:46.059355 28162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5796c9e4af4b49d6b6358dd91aefc8c7" candidate_uuid: "faa8854d1d214547b99cf778591b65c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b723050cd3ee495cb965a2bb373f8bf2" is_pre_election: true
I20260926 20:10:46.059645 28295 raft_consensus.cc:2454] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa8854d1d214547b99cf778591b65c1 in term 0.
I20260926 20:10:46.059645 28162 raft_consensus.cc:2454] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faa8854d1d214547b99cf778591b65c1 in term 0.
I20260926 20:10:46.060221 27962 leader_election.cc:302] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b723050cd3ee495cb965a2bb373f8bf2, faa8854d1d214547b99cf778591b65c1; no voters:
I20260926 20:10:46.060544 28366 raft_consensus.cc:2781] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:46.060647 28366 raft_consensus.cc:482] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:46.060688 28366 raft_consensus.cc:3037] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:46.062140 28366 raft_consensus.cc:504] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa8854d1d214547b99cf778591b65c1" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 34087 } } peers { permanent_uuid: "2fff3239a8cf4483b50f7a10f07b4de0" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41841 } } peers { permanent_uuid: "b723050cd3ee495cb965a2bb373f8bf2" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43535 } }
I20260926 20:10:46.062505 28366 leader_election.cc:288] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [CANDIDATE]: Term 1 election: Requested vote from peers 2fff3239a8cf4483b50f7a10f07b4de0 (127.23.50.131:41841), b723050cd3ee495cb965a2bb373f8bf2 (127.23.50.130:43535)
I20260926 20:10:46.067227 28295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5796c9e4af4b49d6b6358dd91aefc8c7" candidate_uuid: "faa8854d1d214547b99cf778591b65c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fff3239a8cf4483b50f7a10f07b4de0"
I20260926 20:10:46.067262 28162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5796c9e4af4b49d6b6358dd91aefc8c7" candidate_uuid: "faa8854d1d214547b99cf778591b65c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b723050cd3ee495cb965a2bb373f8bf2"
I20260926 20:10:46.067396 28162 raft_consensus.cc:3037] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:46.067397 28295 raft_consensus.cc:3037] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:46.068835 28162 raft_consensus.cc:2454] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa8854d1d214547b99cf778591b65c1 in term 1.
I20260926 20:10:46.068835 28295 raft_consensus.cc:2454] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faa8854d1d214547b99cf778591b65c1 in term 1.
I20260926 20:10:46.069190 27965 leader_election.cc:302] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fff3239a8cf4483b50f7a10f07b4de0, faa8854d1d214547b99cf778591b65c1; no voters:
I20260926 20:10:46.069376 28366 raft_consensus.cc:2781] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:46.069593 28366 raft_consensus.cc:686] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [term 1 LEADER]: Becoming Leader. State: Replica: faa8854d1d214547b99cf778591b65c1, State: Running, Role: LEADER
I20260926 20:10:46.070006 28366 consensus_queue.cc:237] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa8854d1d214547b99cf778591b65c1" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 34087 } } peers { permanent_uuid: "2fff3239a8cf4483b50f7a10f07b4de0" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41841 } } peers { permanent_uuid: "b723050cd3ee495cb965a2bb373f8bf2" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43535 } }
I20260926 20:10:46.073244 27881 catalog_manager.cc:5495] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to faa8854d1d214547b99cf778591b65c1 (127.23.50.129). New cstate: current_term: 1 leader_uuid: "faa8854d1d214547b99cf778591b65c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faa8854d1d214547b99cf778591b65c1" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 34087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fff3239a8cf4483b50f7a10f07b4de0" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b723050cd3ee495cb965a2bb373f8bf2" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43535 } health_report { overall_health: UNKNOWN } } }
I20260926 20:10:46.087850 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.084076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.23.50.129:34087)
0926 20:10:46.084204 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:46.084207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:46.084220 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:10:46.084347 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:46.084349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:46.084401 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:46.084489 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:46.084794 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.085314 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.086273 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:46.086487 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:46.086580 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:46.086924 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:46.087475 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:46.087568 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:46.087688 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:10:46.087742 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:10:46.093835 28162 raft_consensus.cc:1261] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2 [term 1 FOLLOWER]: Refusing update from remote peer faa8854d1d214547b99cf778591b65c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:46.093828 28295 raft_consensus.cc:1261] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0 [term 1 FOLLOWER]: Refusing update from remote peer faa8854d1d214547b99cf778591b65c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:46.094586 28366 consensus_queue.cc:1035] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fff3239a8cf4483b50f7a10f07b4de0" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:46.094790 28369 consensus_queue.cc:1035] T 5796c9e4af4b49d6b6358dd91aefc8c7 P faa8854d1d214547b99cf778591b65c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b723050cd3ee495cb965a2bb373f8bf2" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:46.101969 28372 mvcc.cc:205] Tried to move back new op lower bound from 7333697315193057280 to 7333697315105869824. Current Snapshot: MvccSnapshot[applied={T|T < 7333697315193057280}]
I20260926 20:10:46.102870 28374 mvcc.cc:205] Tried to move back new op lower bound from 7333697315193057280 to 7333697315105869824. Current Snapshot: MvccSnapshot[applied={T|T < 7333697315193057280}]
I20260926 20:10:46.110670 27882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:46.110867 27882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:46.113234 27882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b637dcfb729b4b1e89e9ed1f89499796: Sending DeleteTablet for 3 replicas of tablet 5796c9e4af4b49d6b6358dd91aefc8c7
I20260926 20:10:46.114055 28275 tablet_service.cc:1514] Processing DeleteTablet for tablet 5796c9e4af4b49d6b6358dd91aefc8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:46 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:59890
I20260926 20:10:46.114069 28009 tablet_service.cc:1514] Processing DeleteTablet for tablet 5796c9e4af4b49d6b6358dd91aefc8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:46 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:40600
I20260926 20:10:46.114661 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 27947
I20260926 20:10:46.114708 28142 tablet_service.cc:1514] Processing DeleteTablet for tablet 5796c9e4af4b49d6b6358dd91aefc8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:46 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:58008
I20260926 20:10:46.115460 28383 tablet_replica.cc:330] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2: stopping tablet replica
I20260926 20:10:46.115649 28382 tablet_replica.cc:330] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0: stopping tablet replica
I20260926 20:10:46.115813 28383 raft_consensus.cc:2229] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:46.115932 28382 raft_consensus.cc:2229] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:46.116132 28383 raft_consensus.cc:2258] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:46.116232 28382 raft_consensus.cc:2258] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:46.117645 28382 ts_tablet_manager.cc:1905] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:46.119935 28382 ts_tablet_manager.cc:1918] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:10:46.120088 28382 log.cc:1196] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/5796c9e4af4b49d6b6358dd91aefc8c7
I20260926 20:10:46.120427 28382 ts_tablet_manager.cc:1939] T 5796c9e4af4b49d6b6358dd91aefc8c7 P 2fff3239a8cf4483b50f7a10f07b4de0: Deleting consensus metadata
I20260926 20:10:46.121596 28383 ts_tablet_manager.cc:1905] T 5796c9e4af4b49d6b6358dd91aefc8c7 P b723050cd3ee495cb965a2bb373f8bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:46.121740 27870 catalog_manager.cc:4841] TS 2fff3239a8cf4483b50f7a10f07b4de0 (127.23.50.131:41841): tablet 5796c9e4af4b49d6b6358dd91aefc8c7 (table test-table [id=b129675a837e4cb3ad4e945eff6d6cf3]) successfully deleted
W20260926 20:10:46.122396 27867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:34087 (error 108)
I20260926 20:10:46.122442 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 28080
I20260926 20:10:46.124181 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.123793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:34087 (local address 127.0.0.1:40624)
0926 20:10:46.123994 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:46.124069 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:34087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260926 20:10:46.124347 27867 catalog_manager.cc:4584] TS faa8854d1d214547b99cf778591b65c1 (127.23.50.129:34087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5796c9e4af4b49d6b6358dd91aefc8c7: Network error: Client connection negotiation failed: client connection to 127.23.50.129:34087: connect: Connection refused (error 111)
I20260926 20:10:46.127730 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 28213
I20260926 20:10:46.128744 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.127865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:43535 (local address 127.0.0.1:58024)
0926 20:10:46.128625 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:46.128674 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.130:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":688}
I20260926 20:10:46.134208 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 27852
2026-09-26T20:10:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2288 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T20:10:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:46Z Disabled control of system clock
I20260926 20:10:46.166317 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:44901
--webserver_interface=127.23.50.190
--webserver_port=0
--builtin_ntp_servers=127.23.50.148:33461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:44901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:10:46.286180 28388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:46.286458 28388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:46.286497 28388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:46.290576 28388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:46.290639 28388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:46.290655 28388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:46.290673 28388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:46.290688 28388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:46.295661 28388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:33461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:44901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:44901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:46.296762 28388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:46.298239 28388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:10:46.307192 28388 server_base.cc:1034] running on GCE node
W20260926 20:10:46.307217 28393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:46.307144 28396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:46.307139 28394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:46.308018 28388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:46.309197 28388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:46.310381 28388 hybrid_clock.cc:648] HybridClock initialized: now 1790453446310347 us; error 49 us; skew 500 ppm
I20260926 20:10:46.312418 28388 webserver.cc:460] Webserver started at http://127.23.50.190:33491/ using document root <none> and password file <none>
I20260926 20:10:46.313025 28388 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:46.313119 28388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:46.313369 28388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:46.315636 28388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "17f8ffdf36a14ac48f57c2c42f349127"
format_stamp: "Formatted at 2026-09-26 20:10:46 on dist-test-slave-7034"
server_key: "2298fe81430456757dab14de85eb3724"
server_key_iv: "51a4b50d0e56abd2559136432887364b"
server_key_version: "encryptionkey@0"
I20260926 20:10:46.316236 28388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "17f8ffdf36a14ac48f57c2c42f349127"
format_stamp: "Formatted at 2026-09-26 20:10:46 on dist-test-slave-7034"
server_key: "2298fe81430456757dab14de85eb3724"
server_key_iv: "51a4b50d0e56abd2559136432887364b"
server_key_version: "encryptionkey@0"
I20260926 20:10:46.320158 28388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 20:10:46.323011 28402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:46.324497 28388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:46.324666 28388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "17f8ffdf36a14ac48f57c2c42f349127"
format_stamp: "Formatted at 2026-09-26 20:10:46 on dist-test-slave-7034"
server_key: "2298fe81430456757dab14de85eb3724"
server_key_iv: "51a4b50d0e56abd2559136432887364b"
server_key_version: "encryptionkey@0"
I20260926 20:10:46.324810 28388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:46.329658 28388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:46.330397 28388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:46.330590 28388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:46.337925 28388 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:44901
I20260926 20:10:46.337949 28454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:44901 every 8 connection(s)
I20260926 20:10:46.338901 28388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:46.342438 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 28388
I20260926 20:10:46.342604 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:46.342722 28455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:46.342928 23754 external_mini_cluster.cc:1428] Setting key 08b2d4ab692e7c5f57813ef4afc11d0e
I20260926 20:10:46.349082 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.344424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.23.50.190:44901)
0926 20:10:46.345125 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:46.345141 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:46.345177 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:10:46.346207 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:46.346227 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:46.346602 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:46.346966 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:46.347783 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:46.347788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:46.347800 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:46.347810 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:46.347866 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:46.347882 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:46.348049 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:46.348140 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 20:10:46.348393 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":147,"threads_started":1}
I20260926 20:10:46.350174 28455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127: Bootstrap starting.
I20260926 20:10:46.352519 28455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:46.353307 28455 log.cc:824] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:46.355342 28455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127: No bootstrap required, opened a new log
I20260926 20:10:46.359041 28455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f8ffdf36a14ac48f57c2c42f349127" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 44901 } }
I20260926 20:10:46.359311 28455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:46.359393 28455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f8ffdf36a14ac48f57c2c42f349127, State: Initialized, Role: FOLLOWER
I20260926 20:10:46.359855 28455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f8ffdf36a14ac48f57c2c42f349127" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 44901 } }
I20260926 20:10:46.360008 28455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:46.360100 28455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:46.360208 28455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:46.379333 28455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f8ffdf36a14ac48f57c2c42f349127" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 44901 } }
I20260926 20:10:46.379966 28455 leader_election.cc:302] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17f8ffdf36a14ac48f57c2c42f349127; no voters:
I20260926 20:10:46.380438 28455 leader_election.cc:288] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:46.380754 28460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:46.381081 28460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [term 1 LEADER]: Becoming Leader. State: Replica: 17f8ffdf36a14ac48f57c2c42f349127, State: Running, Role: LEADER
I20260926 20:10:46.381488 28460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f8ffdf36a14ac48f57c2c42f349127" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 44901 } }
I20260926 20:10:46.382174 28455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:46.383240 28461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17f8ffdf36a14ac48f57c2c42f349127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f8ffdf36a14ac48f57c2c42f349127" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 44901 } } }
I20260926 20:10:46.383380 28461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:46.383826 28462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17f8ffdf36a14ac48f57c2c42f349127. Latest consensus state: current_term: 1 leader_uuid: "17f8ffdf36a14ac48f57c2c42f349127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17f8ffdf36a14ac48f57c2c42f349127" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 44901 } } }
I20260926 20:10:46.384001 28462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:46.384974 28469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:46.388146 28469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:46.394162 28469 catalog_manager.cc:1347] Generated new cluster ID: 0691c9943cc7410c92c9b4f0250d7c8b
I20260926 20:10:46.394248 28469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:46.423200 28469 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:46.423615 28469 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:46.428013 28469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127: Generated new TSK 0
I20260926 20:10:46.428856 28469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:10:46.437966 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:44901
--builtin_ntp_servers=127.23.50.148:33461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:10:46.573930 28479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:46.574234 28479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:46.574275 28479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:46.578785 28479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:46.578863 28479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:46.578925 28479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:10:46.584576 28479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:33461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:46.586375 28479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:46.588476 28479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:46.598804 28484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:46.598800 28485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:46.598804 28487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:46.599246 28479 server_base.cc:1034] running on GCE node
I20260926 20:10:46.599732 28479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:46.600432 28479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:46.601650 28479 hybrid_clock.cc:648] HybridClock initialized: now 1790453446601631 us; error 33 us; skew 500 ppm
I20260926 20:10:46.603886 28479 webserver.cc:460] Webserver started at http://127.23.50.129:35969/ using document root <none> and password file <none>
I20260926 20:10:46.604540 28479 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:46.604619 28479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:46.605109 28479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:46.607568 28479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "afb04aa3892e49869ebaef78ecc11142"
format_stamp: "Formatted at 2026-09-26 20:10:46 on dist-test-slave-7034"
server_key: "01da5d24ed1ee0167406607ef97f1c92"
server_key_iv: "b5cb3a8856237a9314fc83792b50f4f0"
server_key_version: "encryptionkey@0"
I20260926 20:10:46.608201 28479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "afb04aa3892e49869ebaef78ecc11142"
format_stamp: "Formatted at 2026-09-26 20:10:46 on dist-test-slave-7034"
server_key: "01da5d24ed1ee0167406607ef97f1c92"
server_key_iv: "b5cb3a8856237a9314fc83792b50f4f0"
server_key_version: "encryptionkey@0"
I20260926 20:10:46.612313 28479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:10:46.615242 28493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:46.616724 28479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:10:46.616870 28479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "afb04aa3892e49869ebaef78ecc11142"
format_stamp: "Formatted at 2026-09-26 20:10:46 on dist-test-slave-7034"
server_key: "01da5d24ed1ee0167406607ef97f1c92"
server_key_iv: "b5cb3a8856237a9314fc83792b50f4f0"
server_key_version: "encryptionkey@0"
I20260926 20:10:46.617002 28479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:46.622310 28479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:46.622998 28479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:46.623347 28479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:46.624025 28479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:46.625141 28479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:46.625229 28479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:46.625305 28479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:46.625349 28479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:46.635110 28479 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:36941
I20260926 20:10:46.635135 28606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:36941 every 8 connection(s)
I20260926 20:10:46.636081 28479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:10:46.642683 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.638088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:49529 (local address 127.23.50.190:44901)
0926 20:10:46.638251 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:46.638256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:46.639476 (+ 1220us) server_negotiation.cc:407] Connection header received
0926 20:10:46.640775 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:46.640778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:46.640820 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:46.640909 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:46.642269 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:46.642272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:46.642274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:46.642288 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:46.642311 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:46.642320 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:46.642405 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:46.642525 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 20:10:46.642567 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:10:46.643615 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.638575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:44901 (local address 127.23.50.129:49529)
0926 20:10:46.639309 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:46.639352 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:46.640547 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:46.641115 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:46.641122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:46.641225 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:46.641973 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:10:46.641975 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:46.642020 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:10:46.642091 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:46.642400 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:46.642407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:46.642411 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:10:46.642545 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 20:10:46.642755 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
I20260926 20:10:46.644956 28607 heartbeater.cc:347] Connected to a master server at 127.23.50.190:44901
I20260926 20:10:46.645265 28607 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:46.645773 28607 heartbeater.cc:511] Master 127.23.50.190:44901 requested a full tablet report, sending...
I20260926 20:10:46.645772 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 28479
I20260926 20:10:46.645993 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:10:46.646306 23754 external_mini_cluster.cc:1428] Setting key 2bf0770ec734ca3c5e2c4a54d35536b8
I20260926 20:10:46.647728 28419 ts_manager.cc:194] Registered new tserver with Master: afb04aa3892e49869ebaef78ecc11142 (127.23.50.129:36941)
I20260926 20:10:46.648680 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:44901
--builtin_ntp_servers=127.23.50.148:33461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 20:10:46.764886 28610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:46.765194 28610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:46.765239 28610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:46.769693 28610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:46.769783 28610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:46.769877 28610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:10:46.775143 28610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:33461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:46.776419 28610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:46.777918 28610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:46.788513 28615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:46.788546 28618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:46.788538 28616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:46.788663 28610 server_base.cc:1034] running on GCE node
I20260926 20:10:46.789288 28610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:46.790091 28610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:46.791292 28610 hybrid_clock.cc:648] HybridClock initialized: now 1790453446791270 us; error 34 us; skew 500 ppm
I20260926 20:10:46.793399 28610 webserver.cc:460] Webserver started at http://127.23.50.130:32897/ using document root <none> and password file <none>
I20260926 20:10:46.794075 28610 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:46.794157 28610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:46.794341 28610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:46.796576 28610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "3530fe94a4834c15a70beddeb2f70b04"
format_stamp: "Formatted at 2026-09-26 20:10:46 on dist-test-slave-7034"
server_key: "3a1002f1545e513b6f5ba5ac5f322b02"
server_key_iv: "5a443949a001d92b32f2166f69754106"
server_key_version: "encryptionkey@0"
I20260926 20:10:46.797138 28610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "3530fe94a4834c15a70beddeb2f70b04"
format_stamp: "Formatted at 2026-09-26 20:10:46 on dist-test-slave-7034"
server_key: "3a1002f1545e513b6f5ba5ac5f322b02"
server_key_iv: "5a443949a001d92b32f2166f69754106"
server_key_version: "encryptionkey@0"
I20260926 20:10:46.801131 28610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 20:10:46.804020 28624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:46.805538 28610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:10:46.805660 28610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "3530fe94a4834c15a70beddeb2f70b04"
format_stamp: "Formatted at 2026-09-26 20:10:46 on dist-test-slave-7034"
server_key: "3a1002f1545e513b6f5ba5ac5f322b02"
server_key_iv: "5a443949a001d92b32f2166f69754106"
server_key_version: "encryptionkey@0"
I20260926 20:10:46.805776 28610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:46.810640 28610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:46.811246 28610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:46.811388 28610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:46.811875 28610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:46.812762 28610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:46.812808 28610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:46.812847 28610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:46.812862 28610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:46.822278 28610 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:44851
I20260926 20:10:46.822305 28737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:44851 every 8 connection(s)
I20260926 20:10:46.823316 28610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:10:46.825536 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 28610
I20260926 20:10:46.825680 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:10:46.826057 23754 external_mini_cluster.cc:1428] Setting key 103a28db7e747b1145718f8675180128
I20260926 20:10:46.828255 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:44901
--builtin_ntp_servers=127.23.50.148:33461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 20:10:46.830380 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.825324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:55765 (local address 127.23.50.190:44901)
0926 20:10:46.825502 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:46.825509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:46.826962 (+ 1453us) server_negotiation.cc:407] Connection header received
0926 20:10:46.828422 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:46.828425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:46.828469 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:46.828544 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:46.829890 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:46.829894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:46.829897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:46.829911 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:46.829939 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:46.829948 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:46.830024 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:46.830237 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 20:10:46.830276 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:10:46.831581 28740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:46.825891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:44901 (local address 127.23.50.130:55765)
0926 20:10:46.826762 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:46.826821 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:46.828197 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:46.828776 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:46.828785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:46.828884 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:46.829504 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:10:46.829508 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:46.829577 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:10:46.829679 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:46.830036 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:46.830046 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:46.830055 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:10:46.830247 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 20:10:46.830532 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":163,"threads_started":1}
I20260926 20:10:46.832877 28738 heartbeater.cc:347] Connected to a master server at 127.23.50.190:44901
I20260926 20:10:46.833127 28738 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:46.833606 28738 heartbeater.cc:511] Master 127.23.50.190:44901 requested a full tablet report, sending...
I20260926 20:10:46.834882 28419 ts_manager.cc:194] Registered new tserver with Master: 3530fe94a4834c15a70beddeb2f70b04 (127.23.50.130:44851)
W20260926 20:10:46.943970 28741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:46.944317 28741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:46.944393 28741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:46.948570 28741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:46.948696 28741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:46.948822 28741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:10:46.954377 28741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:33461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:46.955556 28741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:46.957058 28741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:46.967041 28746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:46.967041 28747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:46.967034 28749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:46.967492 28741 server_base.cc:1034] running on GCE node
I20260926 20:10:46.967969 28741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:46.968707 28741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:46.969962 28741 hybrid_clock.cc:648] HybridClock initialized: now 1790453446969929 us; error 45 us; skew 500 ppm
I20260926 20:10:46.971921 28741 webserver.cc:460] Webserver started at http://127.23.50.131:34155/ using document root <none> and password file <none>
I20260926 20:10:46.972545 28741 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:46.972643 28741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:46.972883 28741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:46.975121 28741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "b54e026119cb413dbd9cb0cd6f682871"
format_stamp: "Formatted at 2026-09-26 20:10:46 on dist-test-slave-7034"
server_key: "bbb02472d839017e4b90b2727bfba4a9"
server_key_iv: "8cd9ee85371970f36ee9664012c71d4a"
server_key_version: "encryptionkey@0"
I20260926 20:10:46.975713 28741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "b54e026119cb413dbd9cb0cd6f682871"
format_stamp: "Formatted at 2026-09-26 20:10:46 on dist-test-slave-7034"
server_key: "bbb02472d839017e4b90b2727bfba4a9"
server_key_iv: "8cd9ee85371970f36ee9664012c71d4a"
server_key_version: "encryptionkey@0"
I20260926 20:10:46.979876 28741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:10:46.982791 28755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:46.984292 28741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:46.984441 28741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "b54e026119cb413dbd9cb0cd6f682871"
format_stamp: "Formatted at 2026-09-26 20:10:46 on dist-test-slave-7034"
server_key: "bbb02472d839017e4b90b2727bfba4a9"
server_key_iv: "8cd9ee85371970f36ee9664012c71d4a"
server_key_version: "encryptionkey@0"
I20260926 20:10:46.984579 28741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:46.989790 28741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:46.990505 28741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:46.990706 28741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:46.991279 28741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:46.992594 28741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:46.992653 28741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:46.992719 28741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:46.992753 28741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:47.001754 28741 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:41295
I20260926 20:10:47.001775 28868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:41295 every 8 connection(s)
I20260926 20:10:47.002759 28741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:10:47.005522 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 28741
I20260926 20:10:47.005653 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:10:47.005986 23754 external_mini_cluster.cc:1428] Setting key 919a0e58f2132b5461ba985851d18e83
I20260926 20:10:47.009745 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.004811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:38927 (local address 127.23.50.190:44901)
0926 20:10:47.004981 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:47.004986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:47.006202 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 20:10:47.008041 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:47.008045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:47.008087 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:47.008175 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:47.009284 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:47.009287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:47.009289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:47.009299 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:47.009319 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:47.009326 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:47.009397 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:47.009593 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 20:10:47.009629 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:10:47.010818 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.005250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:44901 (local address 127.23.50.131:38927)
0926 20:10:47.006050 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:47.006102 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:47.007815 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:47.008370 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:47.008377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:47.008468 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:47.009049 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:10:47.009050 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:47.009089 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:10:47.009154 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:47.009433 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:47.009442 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:47.009448 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:10:47.009633 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 20:10:47.009894 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":149,"threads_started":1}
I20260926 20:10:47.011919 28869 heartbeater.cc:347] Connected to a master server at 127.23.50.190:44901
I20260926 20:10:47.012133 28869 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:47.012532 28869 heartbeater.cc:511] Master 127.23.50.190:44901 requested a full tablet report, sending...
I20260926 20:10:47.013566 28419 ts_manager.cc:194] Registered new tserver with Master: b54e026119cb413dbd9cb0cd6f682871 (127.23.50.131:41295)
I20260926 20:10:47.020045 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:10:47.023778 28458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.022067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.23.50.190:44901)
0926 20:10:47.022296 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:47.022302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:47.022475 (+ 173us) server_negotiation.cc:407] Connection header received
0926 20:10:47.022607 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:47.022609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:47.022645 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:47.022721 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:47.023396 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:47.023399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:47.023401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:47.023411 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:47.023429 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:47.023437 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:47.023518 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:47.023561 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 20:10:47.023591 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260926 20:10:47.026651 28419 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:10:47.029898 28419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:47.042800 28880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.039001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36941 (local address 127.0.0.1:59164)
0926 20:10:47.039421 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:47.039482 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:47.039592 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:47.040816 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:47.040822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:47.040908 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:47.041687 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:10:47.041689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:47.041712 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:10:47.041789 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:47.042407 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:47.042411 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:47.042423 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 20:10:47.042558 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 20:10:47.042605 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":105,"threads_started":1}
I20260926 20:10:47.042829 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.039112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.23.50.129:36941)
0926 20:10:47.040324 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:47.040338 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:47.040365 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:10:47.040423 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:47.040428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:47.040640 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:47.041229 (+ 589us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:47.042181 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:47.042187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:47.042199 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:47.042209 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:47.042249 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:47.042274 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:47.042399 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:47.042584 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 20:10:47.042660 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":166,"threads_started":1}
I20260926 20:10:47.042958 28884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.039914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.23.50.131:41295)
0926 20:10:47.040353 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:47.040364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:47.040384 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:10:47.040430 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:47.040433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:47.040640 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:47.041229 (+ 589us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:47.042180 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:47.042187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:47.042199 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:47.042209 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:47.042250 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:47.042287 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:47.042707 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:47.042749 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 20:10:47.042806 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":146,"threads_started":1}
I20260926 20:10:47.043061 28882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.039425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:41295 (local address 127.0.0.1:36538)
0926 20:10:47.039978 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:47.039992 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:47.040080 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:47.040816 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:47.040822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:47.040908 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:47.041733 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:10:47.041735 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:47.041742 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:10:47.041787 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:47.042398 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:47.042409 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:47.042419 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:10:47.042529 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:10:47.042614 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260926 20:10:47.044140 28883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.039842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:44851 (local address 127.0.0.1:36870)
0926 20:10:47.040156 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:47.040172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:47.040258 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:47.041296 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:47.041301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:47.041318 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:47.041969 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:10:47.041971 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:47.041981 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:10:47.042009 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:47.043919 (+ 1910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:47.043922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:47.043926 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:10:47.043980 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 20:10:47.044015 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":94,"threads_started":1}
I20260926 20:10:47.044646 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.040496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.23.50.130:44851)
0926 20:10:47.040876 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:47.040885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:47.040905 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:10:47.040951 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:47.040956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:47.041102 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:47.041554 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:47.042479 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:47.042487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:47.042502 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:47.042516 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:47.043770 (+ 1254us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:47.043808 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:47.044422 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:47.044473 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 20:10:47.044529 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":174,"threads_started":1}
I20260926 20:10:47.046610 28803 tablet_service.cc:1467] Processing CreateTablet for tablet 40be6adcadc94c2ba20b2f90fdea5880 (DEFAULT_TABLE table=test-table [id=18ad1eeae2104ffc8768d632e6757b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:47.046619 28541 tablet_service.cc:1467] Processing CreateTablet for tablet 40be6adcadc94c2ba20b2f90fdea5880 (DEFAULT_TABLE table=test-table [id=18ad1eeae2104ffc8768d632e6757b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:47.047135 28672 tablet_service.cc:1467] Processing CreateTablet for tablet 40be6adcadc94c2ba20b2f90fdea5880 (DEFAULT_TABLE table=test-table [id=18ad1eeae2104ffc8768d632e6757b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:47.047868 28803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40be6adcadc94c2ba20b2f90fdea5880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:47.047868 28541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40be6adcadc94c2ba20b2f90fdea5880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:47.048134 28672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40be6adcadc94c2ba20b2f90fdea5880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:47.056581 28887 tablet_bootstrap.cc:492] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142: Bootstrap starting.
I20260926 20:10:47.057756 28886 tablet_bootstrap.cc:492] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04: Bootstrap starting.
I20260926 20:10:47.059379 28887 tablet_bootstrap.cc:654] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:47.059947 28886 tablet_bootstrap.cc:654] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:47.060446 28887 log.cc:824] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:47.060794 28886 log.cc:824] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:47.061338 28888 tablet_bootstrap.cc:492] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871: Bootstrap starting.
I20260926 20:10:47.063407 28886 tablet_bootstrap.cc:492] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04: No bootstrap required, opened a new log
I20260926 20:10:47.063407 28887 tablet_bootstrap.cc:492] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142: No bootstrap required, opened a new log
I20260926 20:10:47.063606 28886 ts_tablet_manager.cc:1397] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 20:10:47.063606 28887 ts_tablet_manager.cc:1397] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 20:10:47.063805 28888 tablet_bootstrap.cc:654] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:47.064669 28888 log.cc:824] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:47.066659 28888 tablet_bootstrap.cc:492] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871: No bootstrap required, opened a new log
I20260926 20:10:47.066885 28888 ts_tablet_manager.cc:1397] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:10:47.068339 28886 raft_consensus.cc:348] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb04aa3892e49869ebaef78ecc11142" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36941 } } peers { permanent_uuid: "3530fe94a4834c15a70beddeb2f70b04" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 44851 } } peers { permanent_uuid: "b54e026119cb413dbd9cb0cd6f682871" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41295 } }
I20260926 20:10:47.068374 28887 raft_consensus.cc:348] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb04aa3892e49869ebaef78ecc11142" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36941 } } peers { permanent_uuid: "3530fe94a4834c15a70beddeb2f70b04" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 44851 } } peers { permanent_uuid: "b54e026119cb413dbd9cb0cd6f682871" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41295 } }
I20260926 20:10:47.068620 28886 raft_consensus.cc:374] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:47.068620 28887 raft_consensus.cc:374] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:47.068678 28887 raft_consensus.cc:729] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb04aa3892e49869ebaef78ecc11142, State: Initialized, Role: FOLLOWER
I20260926 20:10:47.068678 28886 raft_consensus.cc:729] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3530fe94a4834c15a70beddeb2f70b04, State: Initialized, Role: FOLLOWER
I20260926 20:10:47.069159 28886 consensus_queue.cc:260] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb04aa3892e49869ebaef78ecc11142" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36941 } } peers { permanent_uuid: "3530fe94a4834c15a70beddeb2f70b04" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 44851 } } peers { permanent_uuid: "b54e026119cb413dbd9cb0cd6f682871" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41295 } }
I20260926 20:10:47.069159 28887 consensus_queue.cc:260] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb04aa3892e49869ebaef78ecc11142" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36941 } } peers { permanent_uuid: "3530fe94a4834c15a70beddeb2f70b04" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 44851 } } peers { permanent_uuid: "b54e026119cb413dbd9cb0cd6f682871" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41295 } }
I20260926 20:10:47.069978 28738 heartbeater.cc:503] Master 127.23.50.190:44901 was elected leader, sending a full tablet report...
I20260926 20:10:47.070214 28886 ts_tablet_manager.cc:1428] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260926 20:10:47.070214 28887 ts_tablet_manager.cc:1428] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 20:10:47.070324 28607 heartbeater.cc:503] Master 127.23.50.190:44901 was elected leader, sending a full tablet report...
I20260926 20:10:47.071630 28888 raft_consensus.cc:348] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb04aa3892e49869ebaef78ecc11142" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36941 } } peers { permanent_uuid: "3530fe94a4834c15a70beddeb2f70b04" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 44851 } } peers { permanent_uuid: "b54e026119cb413dbd9cb0cd6f682871" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41295 } }
I20260926 20:10:47.071924 28888 raft_consensus.cc:374] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:47.071995 28888 raft_consensus.cc:729] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b54e026119cb413dbd9cb0cd6f682871, State: Initialized, Role: FOLLOWER
I20260926 20:10:47.072476 28888 consensus_queue.cc:260] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb04aa3892e49869ebaef78ecc11142" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36941 } } peers { permanent_uuid: "3530fe94a4834c15a70beddeb2f70b04" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 44851 } } peers { permanent_uuid: "b54e026119cb413dbd9cb0cd6f682871" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41295 } }
I20260926 20:10:47.073235 28869 heartbeater.cc:503] Master 127.23.50.190:44901 was elected leader, sending a full tablet report...
I20260926 20:10:47.073688 28888 ts_tablet_manager.cc:1428] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260926 20:10:47.074240 28739 tablet.cc:2369] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:10:47.137359 28608 tablet.cc:2369] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:10:47.175957 28893 raft_consensus.cc:482] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:47.176270 28893 raft_consensus.cc:504] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb04aa3892e49869ebaef78ecc11142" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36941 } } peers { permanent_uuid: "3530fe94a4834c15a70beddeb2f70b04" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 44851 } } peers { permanent_uuid: "b54e026119cb413dbd9cb0cd6f682871" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41295 } }
I20260926 20:10:47.177560 28893 leader_election.cc:288] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3530fe94a4834c15a70beddeb2f70b04 (127.23.50.130:44851), b54e026119cb413dbd9cb0cd6f682871 (127.23.50.131:41295)
I20260926 20:10:47.179978 28884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.177985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:36053 (local address 127.23.50.131:41295)
0926 20:10:47.178133 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:47.178138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:47.178322 (+ 184us) server_negotiation.cc:407] Connection header received
0926 20:10:47.178464 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:47.178467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:47.178511 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:47.178592 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:47.179543 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:47.179547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:47.179550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:47.179563 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:47.179585 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:47.179596 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:47.179698 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:47.179805 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 20:10:47.179844 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:10:47.179978 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.177967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:39525 (local address 127.23.50.130:44851)
0926 20:10:47.178114 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:47.178121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:47.178371 (+ 250us) server_negotiation.cc:407] Connection header received
0926 20:10:47.178479 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:47.178482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:47.178513 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:47.178592 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:47.179543 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:47.179547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:47.179550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:47.179563 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:47.179585 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:47.179596 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:47.179698 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:47.179800 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 20:10:47.179844 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:10:47.180022 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.177838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:41295 (local address 127.23.50.129:36053)
0926 20:10:47.178213 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:47.178228 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:47.178346 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:47.178632 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:47.178635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:47.178652 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:47.179326 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:10:47.179328 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:47.179341 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:10:47.179375 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:47.179700 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:47.179703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:47.179706 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:10:47.179800 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:10:47.179849 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":101,"threads_started":1}
I20260926 20:10:47.180025 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.177863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:44851 (local address 127.23.50.129:39525)
0926 20:10:47.178303 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:47.178319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:47.178387 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:47.178632 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:47.178635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:47.178652 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:47.179326 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:10:47.179328 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:47.179342 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:10:47.179375 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:47.179700 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:47.179703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:47.179706 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:10:47.179792 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 20:10:47.179858 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":196,"thread_start_us":73,"threads_started":1}
I20260926 20:10:47.180728 28823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40be6adcadc94c2ba20b2f90fdea5880" candidate_uuid: "afb04aa3892e49869ebaef78ecc11142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54e026119cb413dbd9cb0cd6f682871" is_pre_election: true
I20260926 20:10:47.181083 28823 raft_consensus.cc:2454] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afb04aa3892e49869ebaef78ecc11142 in term 0.
I20260926 20:10:47.181062 28692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40be6adcadc94c2ba20b2f90fdea5880" candidate_uuid: "afb04aa3892e49869ebaef78ecc11142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3530fe94a4834c15a70beddeb2f70b04" is_pre_election: true
I20260926 20:10:47.181361 28692 raft_consensus.cc:2454] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afb04aa3892e49869ebaef78ecc11142 in term 0.
I20260926 20:10:47.181591 28496 leader_election.cc:302] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afb04aa3892e49869ebaef78ecc11142, b54e026119cb413dbd9cb0cd6f682871; no voters:
I20260926 20:10:47.181914 28893 raft_consensus.cc:2781] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:47.182000 28893 raft_consensus.cc:482] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:47.182044 28893 raft_consensus.cc:3037] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:47.183135 28893 raft_consensus.cc:504] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb04aa3892e49869ebaef78ecc11142" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36941 } } peers { permanent_uuid: "3530fe94a4834c15a70beddeb2f70b04" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 44851 } } peers { permanent_uuid: "b54e026119cb413dbd9cb0cd6f682871" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41295 } }
I20260926 20:10:47.183522 28893 leader_election.cc:288] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [CANDIDATE]: Term 1 election: Requested vote from peers 3530fe94a4834c15a70beddeb2f70b04 (127.23.50.130:44851), b54e026119cb413dbd9cb0cd6f682871 (127.23.50.131:41295)
I20260926 20:10:47.183910 28823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40be6adcadc94c2ba20b2f90fdea5880" candidate_uuid: "afb04aa3892e49869ebaef78ecc11142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b54e026119cb413dbd9cb0cd6f682871"
I20260926 20:10:47.183898 28692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40be6adcadc94c2ba20b2f90fdea5880" candidate_uuid: "afb04aa3892e49869ebaef78ecc11142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3530fe94a4834c15a70beddeb2f70b04"
I20260926 20:10:47.184046 28692 raft_consensus.cc:3037] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:47.184046 28823 raft_consensus.cc:3037] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:47.185472 28823 raft_consensus.cc:2454] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb04aa3892e49869ebaef78ecc11142 in term 1.
I20260926 20:10:47.185472 28692 raft_consensus.cc:2454] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb04aa3892e49869ebaef78ecc11142 in term 1.
I20260926 20:10:47.185765 28496 leader_election.cc:302] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afb04aa3892e49869ebaef78ecc11142, b54e026119cb413dbd9cb0cd6f682871; no voters:
I20260926 20:10:47.185963 28893 raft_consensus.cc:2781] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:47.186168 28893 raft_consensus.cc:686] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [term 1 LEADER]: Becoming Leader. State: Replica: afb04aa3892e49869ebaef78ecc11142, State: Running, Role: LEADER
I20260926 20:10:47.186537 28893 consensus_queue.cc:237] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb04aa3892e49869ebaef78ecc11142" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36941 } } peers { permanent_uuid: "3530fe94a4834c15a70beddeb2f70b04" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 44851 } } peers { permanent_uuid: "b54e026119cb413dbd9cb0cd6f682871" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41295 } }
I20260926 20:10:47.189958 28419 catalog_manager.cc:5495] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 reported cstate change: term changed from 0 to 1, leader changed from <none> to afb04aa3892e49869ebaef78ecc11142 (127.23.50.129). New cstate: current_term: 1 leader_uuid: "afb04aa3892e49869ebaef78ecc11142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb04aa3892e49869ebaef78ecc11142" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3530fe94a4834c15a70beddeb2f70b04" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b54e026119cb413dbd9cb0cd6f682871" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41295 } health_report { overall_health: UNKNOWN } } }
I20260926 20:10:47.199383 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.197685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.23.50.129:36941)
0926 20:10:47.197808 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:47.197811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:47.197824 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:10:47.198019 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:47.198021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:47.198083 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:47.198194 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:47.198929 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:47.198932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:47.198934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:47.198950 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:47.198971 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:47.198979 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:47.199060 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:47.199206 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 20:10:47.199247 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:10:47.205726 28823 raft_consensus.cc:1261] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871 [term 1 FOLLOWER]: Refusing update from remote peer afb04aa3892e49869ebaef78ecc11142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:47.205726 28692 raft_consensus.cc:1261] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04 [term 1 FOLLOWER]: Refusing update from remote peer afb04aa3892e49869ebaef78ecc11142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:47.206789 28897 consensus_queue.cc:1035] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b54e026119cb413dbd9cb0cd6f682871" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 41295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:47.207038 28893 consensus_queue.cc:1035] T 40be6adcadc94c2ba20b2f90fdea5880 P afb04aa3892e49869ebaef78ecc11142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3530fe94a4834c15a70beddeb2f70b04" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:47.213934 28900 mvcc.cc:205] Tried to move back new op lower bound from 7333697319745961984 to 7333697319678959616. Current Snapshot: MvccSnapshot[applied={T|T < 7333697319745961984}]
I20260926 20:10:47.217658 28902 mvcc.cc:205] Tried to move back new op lower bound from 7333697319745961984 to 7333697319678959616. Current Snapshot: MvccSnapshot[applied={T|T < 7333697319745961984}]
I20260926 20:10:47.224587 28419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:47.224797 28419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:47.227615 28419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17f8ffdf36a14ac48f57c2c42f349127: Sending DeleteTablet for 3 replicas of tablet 40be6adcadc94c2ba20b2f90fdea5880
I20260926 20:10:47.228618 28672 tablet_service.cc:1514] Processing DeleteTablet for tablet 40be6adcadc94c2ba20b2f90fdea5880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:47 UTC) from {username='slave'} at 127.0.0.1:36870
I20260926 20:10:47.228703 28803 tablet_service.cc:1514] Processing DeleteTablet for tablet 40be6adcadc94c2ba20b2f90fdea5880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:47 UTC) from {username='slave'} at 127.0.0.1:36538
I20260926 20:10:47.229211 28910 tablet_replica.cc:330] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04: stopping tablet replica
I20260926 20:10:47.229204 28541 tablet_service.cc:1514] Processing DeleteTablet for tablet 40be6adcadc94c2ba20b2f90fdea5880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:47 UTC) from {username='slave'} at 127.0.0.1:59164
I20260926 20:10:47.229564 28909 tablet_replica.cc:330] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871: stopping tablet replica
I20260926 20:10:47.229581 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 28479
I20260926 20:10:47.230285 28910 raft_consensus.cc:2229] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:47.230404 28909 raft_consensus.cc:2229] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:47.230803 28910 raft_consensus.cc:2258] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:47.231304 28909 raft_consensus.cc:2258] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:47.232713 28910 ts_tablet_manager.cc:1905] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:47.233202 28909 ts_tablet_manager.cc:1905] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:47.235702 28910 ts_tablet_manager.cc:1918] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:10:47.235843 28910 log.cc:1196] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/40be6adcadc94c2ba20b2f90fdea5880
I20260926 20:10:47.235941 28909 ts_tablet_manager.cc:1918] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:10:47.236048 28909 log.cc:1196] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/40be6adcadc94c2ba20b2f90fdea5880
I20260926 20:10:47.236204 28910 ts_tablet_manager.cc:1939] T 40be6adcadc94c2ba20b2f90fdea5880 P 3530fe94a4834c15a70beddeb2f70b04: Deleting consensus metadata
I20260926 20:10:47.236356 28909 ts_tablet_manager.cc:1939] T 40be6adcadc94c2ba20b2f90fdea5880 P b54e026119cb413dbd9cb0cd6f682871: Deleting consensus metadata
W20260926 20:10:47.236326 28404 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:36941 (error 108)
I20260926 20:10:47.237208 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 28610
I20260926 20:10:47.237978 28405 catalog_manager.cc:4841] TS b54e026119cb413dbd9cb0cd6f682871 (127.23.50.131:41295): tablet 40be6adcadc94c2ba20b2f90fdea5880 (table test-table [id=18ad1eeae2104ffc8768d632e6757b98]) successfully deleted
I20260926 20:10:47.241391 28883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.240990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36941 (local address 127.0.0.1:59172)
0926 20:10:47.241209 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:47.241299 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260926 20:10:47.241649 28404 catalog_manager.cc:4584] TS afb04aa3892e49869ebaef78ecc11142 (127.23.50.129:36941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40be6adcadc94c2ba20b2f90fdea5880: Network error: Client connection negotiation failed: client connection to 127.23.50.129:36941: connect: Connection refused (error 111)
I20260926 20:10:47.245425 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 28741
I20260926 20:10:47.246668 28883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.245739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:44851 (local address 127.0.0.1:36886)
0926 20:10:47.246479 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:47.246552 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.130:44851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617}
I20260926 20:10:47.251338 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 28388
2026-09-26T20:10:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1115 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 20:10:47.273334 23754 mini_oidc.cc:152] Starting JWKS server
I20260926 20:10:47.275172 23754 webserver.cc:460] Webserver started at https://127.0.0.1:36111/ using document root <none> and password file <none>
I20260926 20:10:47.275337 23754 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:10:47.275599 23754 webserver.cc:460] Webserver started at http://127.0.0.1:33837/ using document root <none> and password file <none>
2026-09-26T20:10:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:47Z Disabled control of system clock
I20260926 20:10:47.285704 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:46069
--webserver_interface=127.23.50.190
--webserver_port=0
--builtin_ntp_servers=127.23.50.148:44795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:46069
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36111/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/testchainca.cert with env {}
W20260926 20:10:47.397797 28918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:47.398165 28918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:47.398247 28918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:47.402372 28918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:47.402474 28918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:47.402503 28918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:47.402539 28918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:47.402575 28918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:10:47.402609 28918 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:10:47.402638 28918 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36111/jwks.json
I20260926 20:10:47.408221 28918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:46069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:46069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36111/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:47.409400 28918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:47.410470 28918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:47.419072 28924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:47.419080 28923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:47.419072 28926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:47.419462 28918 server_base.cc:1034] running on GCE node
I20260926 20:10:47.420081 28918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:47.421227 28918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:47.422425 28918 hybrid_clock.cc:648] HybridClock initialized: now 1790453447422411 us; error 43 us; skew 500 ppm
I20260926 20:10:47.424580 28918 webserver.cc:460] Webserver started at http://127.23.50.190:37627/ using document root <none> and password file <none>
I20260926 20:10:47.425228 28918 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:47.425297 28918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:47.425530 28918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:47.427765 28918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "597798bd676b47129b27e1158831670d"
format_stamp: "Formatted at 2026-09-26 20:10:47 on dist-test-slave-7034"
server_key: "b7ca7a5bf79c04aeda710f1253dc6c81"
server_key_iv: "4f62b1b13397600f1b1848dd7c8625cf"
server_key_version: "encryptionkey@0"
I20260926 20:10:47.428378 28918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "597798bd676b47129b27e1158831670d"
format_stamp: "Formatted at 2026-09-26 20:10:47 on dist-test-slave-7034"
server_key: "b7ca7a5bf79c04aeda710f1253dc6c81"
server_key_iv: "4f62b1b13397600f1b1848dd7c8625cf"
server_key_version: "encryptionkey@0"
I20260926 20:10:47.432293 28918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:10:47.434878 28932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:47.436290 28918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:47.436458 28918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "597798bd676b47129b27e1158831670d"
format_stamp: "Formatted at 2026-09-26 20:10:47 on dist-test-slave-7034"
server_key: "b7ca7a5bf79c04aeda710f1253dc6c81"
server_key_iv: "4f62b1b13397600f1b1848dd7c8625cf"
server_key_version: "encryptionkey@0"
I20260926 20:10:47.436587 28918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:47.463835 28918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:47.464622 28918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:47.464845 28918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:47.472297 28918 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:46069
I20260926 20:10:47.472321 28984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:46069 every 8 connection(s)
I20260926 20:10:47.473250 28918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:47.476703 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:47.482168 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 28918
I20260926 20:10:47.482311 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:47.482578 23754 external_mini_cluster.cc:1428] Setting key 9de05071ddb62e84f05b253879f646ab
I20260926 20:10:47.482632 28985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d: Bootstrap starting.
I20260926 20:10:47.485325 28985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:47.486224 28985 log.cc:824] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:47.488170 28985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d: No bootstrap required, opened a new log
I20260926 20:10:47.491861 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.483866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.23.50.190:46069)
0926 20:10:47.484571 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:47.484586 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:47.484621 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:10:47.485268 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:47.485285 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:47.485806 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:47.486200 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:47.486640 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:47.487619 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:47.488622 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:47.489094 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:47.489265 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:47.490124 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:47.490130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:47.490143 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:47.490542 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:47.490597 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:47.490617 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:47.490730 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:47.490848 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 20:10:47.491118 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":153,"threads_started":1}
I20260926 20:10:47.491864 28985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "597798bd676b47129b27e1158831670d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46069 } }
I20260926 20:10:47.492230 28985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:47.492300 28985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597798bd676b47129b27e1158831670d, State: Initialized, Role: FOLLOWER
I20260926 20:10:47.492779 28985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "597798bd676b47129b27e1158831670d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46069 } }
I20260926 20:10:47.492938 28985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:47.493032 28985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:47.493156 28985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:47.494324 28985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "597798bd676b47129b27e1158831670d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46069 } }
I20260926 20:10:47.494696 28985 leader_election.cc:302] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 597798bd676b47129b27e1158831670d; no voters:
I20260926 20:10:47.494971 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:10:47.495085 28985 leader_election.cc:288] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:47.495170 28990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:47.495340 28990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [term 1 LEADER]: Becoming Leader. State: Replica: 597798bd676b47129b27e1158831670d, State: Running, Role: LEADER
I20260926 20:10:47.495656 28990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "597798bd676b47129b27e1158831670d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46069 } }
I20260926 20:10:47.496397 28985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:47.497385 28991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 597798bd676b47129b27e1158831670d. Latest consensus state: current_term: 1 leader_uuid: "597798bd676b47129b27e1158831670d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "597798bd676b47129b27e1158831670d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46069 } } }
I20260926 20:10:47.497650 28992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "597798bd676b47129b27e1158831670d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "597798bd676b47129b27e1158831670d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46069 } } }
I20260926 20:10:47.497808 28992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:47.497766 28991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:47.498230 28999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:47.501519 28999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:47.508082 28999 catalog_manager.cc:1347] Generated new cluster ID: 87bd80bee03f45b9920e88eb5e8d905d
I20260926 20:10:47.508178 28999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:47.516870 28999 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:47.518187 28999 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:47.527489 28999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 597798bd676b47129b27e1158831670d: Generated new TSK 0
I20260926 20:10:47.528299 28999 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:10:47.562870 29015 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 20:10:47.563000 29015 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:10:47.563546 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.546375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.23.50.190:46069)
0926 20:10:47.546522 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:47.546532 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:47.546689 (+ 157us) server_negotiation.cc:407] Connection header received
0926 20:10:47.546929 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:47.546931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:47.546984 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:47.547074 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:10:47.547394 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:47.547943 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:47.549229 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:47.549508 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:47.549594 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:47.550032 (+ 438us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:10:47.563155 (+ 13123us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:10:47.563430 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52686: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 20:10:47.563658 28988 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52686: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 20:10:47.565127 29014 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:10:47.546195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:46069 (local address 127.0.0.1:52686)
0926 20:10:47.546588 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:47.546605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:47.546756 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:47.547117 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:47.547122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:47.547138 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:10:47.547279 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:47.547286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:47.548059 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:47.548062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:47.549080 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:47.549092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:47.549633 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:47.549637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:47.549738 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:47.549799 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:10:47.563436 (+ 13637us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 20:10:47.563717 (+ 281us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.50.190:46069: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.50.190:46069: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 20:10:47.566008 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 28918
2026-09-26T20:10:47Z chronyd exiting
I20260926 20:10:47.603049 23754 test_util.cc:175] -----------------------------------------------
I20260926 20:10:47.603140 23754 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790453421225356-23754-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (331 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 20:10:47.604215 23754 mini_oidc.cc:152] Starting JWKS server
I20260926 20:10:47.605136 23754 webserver.cc:460] Webserver started at https://127.0.0.1:34993/ using document root <none> and password file <none>
I20260926 20:10:47.605254 23754 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:10:47.605551 23754 webserver.cc:460] Webserver started at http://127.0.0.1:41495/ using document root <none> and password file <none>
2026-09-26T20:10:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:47Z Disabled control of system clock
I20260926 20:10:47.615239 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:43747
--webserver_interface=127.23.50.190
--webserver_port=0
--builtin_ntp_servers=127.23.50.148:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:43747
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34993/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/testchainca.cert with env {}
W20260926 20:10:47.728761 29021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:47.729041 29021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:47.729082 29021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:47.733182 29021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:47.733253 29021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:47.733269 29021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:47.733286 29021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:47.733301 29021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:10:47.733317 29021 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:10:47.733331 29021 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34993/jwks.json
I20260926 20:10:47.738917 29021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:43747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:43747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34993/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:47.739981 29021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:47.740944 29021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:47.749631 29027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:47.749624 29026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:47.749752 29021 server_base.cc:1034] running on GCE node
W20260926 20:10:47.749704 29029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:47.750658 29021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:47.751860 29021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:47.753089 29021 hybrid_clock.cc:648] HybridClock initialized: now 1790453447753057 us; error 48 us; skew 500 ppm
I20260926 20:10:47.755329 29021 webserver.cc:460] Webserver started at http://127.23.50.190:37857/ using document root <none> and password file <none>
I20260926 20:10:47.755976 29021 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:47.756044 29021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:47.756306 29021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:47.758538 29021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "3af77d5a303041a18bf63a42fc9b7f2e"
format_stamp: "Formatted at 2026-09-26 20:10:47 on dist-test-slave-7034"
server_key: "8b80b21715674913a9d8e786a8395c82"
server_key_iv: "720431069b56a46b4c6f6c73c4fa4b27"
server_key_version: "encryptionkey@0"
I20260926 20:10:47.759135 29021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "3af77d5a303041a18bf63a42fc9b7f2e"
format_stamp: "Formatted at 2026-09-26 20:10:47 on dist-test-slave-7034"
server_key: "8b80b21715674913a9d8e786a8395c82"
server_key_iv: "720431069b56a46b4c6f6c73c4fa4b27"
server_key_version: "encryptionkey@0"
I20260926 20:10:47.762869 29021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:10:47.765270 29035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:47.766712 29021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:10:47.766856 29021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "3af77d5a303041a18bf63a42fc9b7f2e"
format_stamp: "Formatted at 2026-09-26 20:10:47 on dist-test-slave-7034"
server_key: "8b80b21715674913a9d8e786a8395c82"
server_key_iv: "720431069b56a46b4c6f6c73c4fa4b27"
server_key_version: "encryptionkey@0"
I20260926 20:10:47.766978 29021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:47.789467 29021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:47.790314 29021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:47.790524 29021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:47.797693 29021 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:43747
I20260926 20:10:47.797713 29087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:43747 every 8 connection(s)
I20260926 20:10:47.798743 29021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:47.801940 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 29021
I20260926 20:10:47.802070 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:47.802273 29088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:47.802392 23754 external_mini_cluster.cc:1428] Setting key a1aa983d3f4d633983f2cdac821376a8
I20260926 20:10:47.809039 29088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e: Bootstrap starting.
I20260926 20:10:47.811546 29088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:47.812212 29091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.803833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.23.50.190:43747)
0926 20:10:47.804521 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:47.804536 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:47.804570 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:10:47.805560 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:47.805581 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:47.806067 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:47.806430 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:47.806867 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:47.807840 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:47.808810 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:47.809358 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:47.809521 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:47.810474 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:47.810480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:47.810492 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:47.810890 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:47.810947 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:47.810967 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:47.811075 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:47.811191 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 20:10:47.811467 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":160,"threads_started":1}
I20260926 20:10:47.812299 29088 log.cc:824] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:47.814584 29088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e: No bootstrap required, opened a new log
I20260926 20:10:47.815290 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:10:47.819010 29088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af77d5a303041a18bf63a42fc9b7f2e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43747 } }
I20260926 20:10:47.819254 29088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:47.819331 29088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af77d5a303041a18bf63a42fc9b7f2e, State: Initialized, Role: FOLLOWER
I20260926 20:10:47.819743 29088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af77d5a303041a18bf63a42fc9b7f2e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43747 } }
I20260926 20:10:47.819897 29088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:47.819965 29088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:47.820063 29088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:47.821013 29088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af77d5a303041a18bf63a42fc9b7f2e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43747 } }
I20260926 20:10:47.821381 29088 leader_election.cc:302] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3af77d5a303041a18bf63a42fc9b7f2e; no voters:
I20260926 20:10:47.821695 29088 leader_election.cc:288] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:47.821851 29093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:47.822146 29093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [term 1 LEADER]: Becoming Leader. State: Replica: 3af77d5a303041a18bf63a42fc9b7f2e, State: Running, Role: LEADER
I20260926 20:10:47.822463 29093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af77d5a303041a18bf63a42fc9b7f2e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43747 } }
I20260926 20:10:47.823037 29088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:47.824174 29094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3af77d5a303041a18bf63a42fc9b7f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af77d5a303041a18bf63a42fc9b7f2e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43747 } } }
I20260926 20:10:47.824307 29094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:47.824625 29095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3af77d5a303041a18bf63a42fc9b7f2e. Latest consensus state: current_term: 1 leader_uuid: "3af77d5a303041a18bf63a42fc9b7f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3af77d5a303041a18bf63a42fc9b7f2e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43747 } } }
I20260926 20:10:47.824724 29095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:47.824826 29102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:47.827651 29102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:47.832969 29102 catalog_manager.cc:1347] Generated new cluster ID: 0f037e53ef38475996bf44548a887eb8
I20260926 20:10:47.833036 29102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:47.840711 29102 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:47.841596 29102 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:47.848091 29102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3af77d5a303041a18bf63a42fc9b7f2e: Generated new TSK 0
I20260926 20:10:47.848934 29102 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 20:10:47.879877 29118 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 20:10:47.880080 29118 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 20:10:47.880725 29091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:47.863479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.23.50.190:43747)
0926 20:10:47.863619 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:47.863624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:47.863761 (+ 137us) server_negotiation.cc:407] Connection header received
0926 20:10:47.863931 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:47.863934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:47.863997 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:47.864126 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:10:47.864386 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:47.865244 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:47.866347 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:47.866705 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:47.866816 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:47.867204 (+ 388us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:10:47.880265 (+ 13061us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34993/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:10:47.880561 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34993/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 20:10:47.880801 29091 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34993/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 20:10:47.880873 29117 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:10:47.863378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43747 (local address 127.0.0.1:53634)
0926 20:10:47.863664 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:47.863684 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:47.863786 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:47.864127 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:47.864129 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:47.864136 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:10:47.864278 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:47.864283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:47.865390 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:47.865394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:47.866212 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:47.866224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:47.866859 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:47.866862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:47.866942 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:47.867006 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:10:47.880513 (+ 13507us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34993/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 20:10:47.880678 (+ 165us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.50.190:43747: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34993/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":102,"threads_started":1}
I20260926 20:10:47.881583 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 29021
2026-09-26T20:10:47Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (316 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 20:10:47.920861 23754 mini_oidc.cc:152] Starting JWKS server
I20260926 20:10:47.921979 23754 webserver.cc:460] Webserver started at https://127.0.0.1:40365/ using document root <none> and password file <none>
I20260926 20:10:47.922102 23754 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 20:10:47.922405 23754 webserver.cc:460] Webserver started at http://127.0.0.1:40147/ using document root <none> and password file <none>
2026-09-26T20:10:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:47Z Disabled control of system clock
I20260926 20:10:47.932018 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42651
--webserver_interface=127.23.50.190
--webserver_port=0
--builtin_ntp_servers=127.23.50.148:39629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:42651
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40365/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:10:48.043782 29124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:48.044064 29124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:48.044103 29124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:48.048060 29124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:48.048137 29124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:48.048154 29124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:48.048170 29124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:48.048185 29124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:10:48.048202 29124 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:10:48.048214 29124 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40365/jwks.json
I20260926 20:10:48.053053 29124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:39629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:42651
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40365/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:48.054190 29124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:48.055534 29124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:48.064409 29129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:48.064416 29132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:48.064558 29124 server_base.cc:1034] running on GCE node
W20260926 20:10:48.064512 29130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:48.065387 29124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:48.066633 29124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:48.067829 29124 hybrid_clock.cc:648] HybridClock initialized: now 1790453448067814 us; error 27 us; skew 500 ppm
I20260926 20:10:48.069644 29124 webserver.cc:460] Webserver started at http://127.23.50.190:43899/ using document root <none> and password file <none>
I20260926 20:10:48.070318 29124 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:48.070418 29124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:48.070658 29124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:48.072724 29124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "e9465ed656e843c7924a04109280e6aa"
format_stamp: "Formatted at 2026-09-26 20:10:48 on dist-test-slave-7034"
server_key: "79e48fd25432d913eb386cfc44233417"
server_key_iv: "5a0e82471cb485f5f1c131d0fc215d33"
server_key_version: "encryptionkey@0"
I20260926 20:10:48.073313 29124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "e9465ed656e843c7924a04109280e6aa"
format_stamp: "Formatted at 2026-09-26 20:10:48 on dist-test-slave-7034"
server_key: "79e48fd25432d913eb386cfc44233417"
server_key_iv: "5a0e82471cb485f5f1c131d0fc215d33"
server_key_version: "encryptionkey@0"
I20260926 20:10:48.077085 29124 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 20:10:48.079823 29138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:48.081084 29124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:10:48.081220 29124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "e9465ed656e843c7924a04109280e6aa"
format_stamp: "Formatted at 2026-09-26 20:10:48 on dist-test-slave-7034"
server_key: "79e48fd25432d913eb386cfc44233417"
server_key_iv: "5a0e82471cb485f5f1c131d0fc215d33"
server_key_version: "encryptionkey@0"
I20260926 20:10:48.081369 29124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:48.118677 29124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:48.119477 29124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:48.119693 29124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:48.126741 29124 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:42651
I20260926 20:10:48.126748 29190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:42651 every 8 connection(s)
I20260926 20:10:48.127718 29124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:48.128309 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 29124
I20260926 20:10:48.128438 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:48.128758 23754 external_mini_cluster.cc:1428] Setting key 53cea5f87e18f339c11246d66e091e3d
I20260926 20:10:48.130944 29191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:48.137635 29191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa: Bootstrap starting.
I20260926 20:10:48.138381 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:48.130159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49646 (local address 127.23.50.190:42651)
0926 20:10:48.130830 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:48.130847 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:48.130883 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:10:48.131582 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:48.131599 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:48.132007 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:48.132304 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:48.132646 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:48.133605 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:48.134890 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:48.135490 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:48.135618 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:48.136483 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:48.136488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:48.136501 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:48.136933 (+ 432us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:48.136994 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:48.137015 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:48.137092 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:48.137251 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 20:10:48.137555 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":156,"threads_started":1}
I20260926 20:10:48.140177 29191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:48.141062 29191 log.cc:824] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:48.141390 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:10:48.143273 29191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa: No bootstrap required, opened a new log
I20260926 20:10:48.146783 29191 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9465ed656e843c7924a04109280e6aa" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42651 } }
I20260926 20:10:48.147018 29191 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:48.147099 29191 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9465ed656e843c7924a04109280e6aa, State: Initialized, Role: FOLLOWER
I20260926 20:10:48.147553 29191 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9465ed656e843c7924a04109280e6aa" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42651 } }
I20260926 20:10:48.147692 29191 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:48.147785 29191 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:48.147898 29191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:48.148895 29191 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9465ed656e843c7924a04109280e6aa" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42651 } }
I20260926 20:10:48.149228 29191 leader_election.cc:302] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9465ed656e843c7924a04109280e6aa; no voters:
I20260926 20:10:48.149557 29191 leader_election.cc:288] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:48.149691 29196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:48.149940 29196 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [term 1 LEADER]: Becoming Leader. State: Replica: e9465ed656e843c7924a04109280e6aa, State: Running, Role: LEADER
I20260926 20:10:48.150236 29196 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9465ed656e843c7924a04109280e6aa" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42651 } }
I20260926 20:10:48.150795 29191 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:48.151968 29197 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9465ed656e843c7924a04109280e6aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9465ed656e843c7924a04109280e6aa" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42651 } } }
I20260926 20:10:48.151937 29198 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9465ed656e843c7924a04109280e6aa. Latest consensus state: current_term: 1 leader_uuid: "e9465ed656e843c7924a04109280e6aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9465ed656e843c7924a04109280e6aa" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42651 } } }
I20260926 20:10:48.152213 29197 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:48.152213 29198 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:48.152647 29206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:48.155799 29206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:48.161545 29206 catalog_manager.cc:1347] Generated new cluster ID: aec208e355d7475fb3af3ba677280244
I20260926 20:10:48.161650 29206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:48.171808 29206 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:48.172755 29206 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:48.181497 29206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9465ed656e843c7924a04109280e6aa: Generated new TSK 0
I20260926 20:10:48.182328 29206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:10:48.202764 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:48.197440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49660 (local address 127.23.50.190:42651)
0926 20:10:48.197592 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:48.197597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:48.197816 (+ 219us) server_negotiation.cc:407] Connection header received
0926 20:10:48.198024 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:48.198027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:48.198078 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:48.198173 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:10:48.198449 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:48.198982 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:48.200245 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:48.200471 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:48.200565 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:48.200932 (+ 367us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:10:48.202341 (+ 1409us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:10:48.202615 (+ 274us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49660: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 20:10:48.202859 29194 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49660: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 20:10:48.205576 29220 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:10:48.197390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42651 (local address 127.0.0.1:49660)
0926 20:10:48.197710 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:48.197725 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:48.197876 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:48.198195 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:48.198198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:48.198206 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:10:48.198341 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:48.198348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:48.199089 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:48.199092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:48.200089 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:48.200103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:48.200598 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:48.200602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:48.200697 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:48.200726 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:10:48.202523 (+ 1797us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 20:10:48.205418 (+ 2895us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.50.190:42651: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40365/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":2660,"thread_start_us":131,"threads_started":1}
I20260926 20:10:48.206288 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 29124
E20260926 20:10:48.216852 29221 webserver.cc:533] Webserver: sslize failed:
E20260926 20:10:48.216943 29221 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T20:10:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (325 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T20:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:48Z Disabled control of system clock
I20260926 20:10:48.255453 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37491
--webserver_interface=127.23.50.190
--webserver_port=0
--builtin_ntp_servers=127.23.50.148:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:37491
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/testchainca.cert with env {}
W20260926 20:10:48.368614 29225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:48.368913 29225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:48.368955 29225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:48.373266 29225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:48.373335 29225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:48.373350 29225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:48.373368 29225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:48.373383 29225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 20:10:48.373425 29225 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 20:10:48.373461 29225 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 20:10:48.379045 29225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:37491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:48.380275 29225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:48.381428 29225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:48.390507 29230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:48.390596 29225 server_base.cc:1034] running on GCE node
W20260926 20:10:48.390522 29233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:48.390578 29231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:48.391399 29225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:48.392583 29225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:48.393764 29225 hybrid_clock.cc:648] HybridClock initialized: now 1790453448393732 us; error 50 us; skew 500 ppm
I20260926 20:10:48.395840 29225 webserver.cc:460] Webserver started at http://127.23.50.190:35361/ using document root <none> and password file <none>
I20260926 20:10:48.396445 29225 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:48.396510 29225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:48.396727 29225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:48.398931 29225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "3397a014bc5747e8964b53d3291d0d9f"
format_stamp: "Formatted at 2026-09-26 20:10:48 on dist-test-slave-7034"
server_key: "2e3f46bd46e59c4929efc4d4079fac99"
server_key_iv: "99904dc6fa489b394572de3a0d71e31f"
server_key_version: "encryptionkey@0"
I20260926 20:10:48.399511 29225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "3397a014bc5747e8964b53d3291d0d9f"
format_stamp: "Formatted at 2026-09-26 20:10:48 on dist-test-slave-7034"
server_key: "2e3f46bd46e59c4929efc4d4079fac99"
server_key_iv: "99904dc6fa489b394572de3a0d71e31f"
server_key_version: "encryptionkey@0"
I20260926 20:10:48.403474 29225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:10:48.406224 29239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:48.407526 29225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:10:48.407665 29225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "3397a014bc5747e8964b53d3291d0d9f"
format_stamp: "Formatted at 2026-09-26 20:10:48 on dist-test-slave-7034"
server_key: "2e3f46bd46e59c4929efc4d4079fac99"
server_key_iv: "99904dc6fa489b394572de3a0d71e31f"
server_key_version: "encryptionkey@0"
I20260926 20:10:48.407799 29225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:48.421459 29225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:48.422300 29225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:48.422513 29225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:48.430111 29225 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:37491
I20260926 20:10:48.430181 29291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:37491 every 8 connection(s)
I20260926 20:10:48.431138 29225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:48.431706 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 29225
I20260926 20:10:48.431854 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:48.432169 23754 external_mini_cluster.cc:1428] Setting key 04156c976ccfb66303c5eefe2db586b3
I20260926 20:10:48.434641 29292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:48.441267 29292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f: Bootstrap starting.
I20260926 20:10:48.442420 29295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:48.433607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.23.50.190:37491)
0926 20:10:48.434315 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:48.434333 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:48.434372 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:10:48.435129 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:48.435152 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:48.435576 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:48.435892 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:48.436275 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:48.437392 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:48.438696 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:48.439264 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:48.439396 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:48.440472 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:48.440478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:48.440492 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:10:48.440895 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:48.440955 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:10:48.440974 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:48.441057 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:48.441250 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 20:10:48.441607 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":176,"threads_started":1}
I20260926 20:10:48.443950 29292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:48.444828 29292 log.cc:824] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:48.445523 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:10:48.447260 29292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f: No bootstrap required, opened a new log
I20260926 20:10:48.451323 29292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3397a014bc5747e8964b53d3291d0d9f" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37491 } }
I20260926 20:10:48.451573 29292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:48.451666 29292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3397a014bc5747e8964b53d3291d0d9f, State: Initialized, Role: FOLLOWER
I20260926 20:10:48.452077 29292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3397a014bc5747e8964b53d3291d0d9f" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37491 } }
I20260926 20:10:48.452195 29292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:48.452241 29292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:48.452317 29292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:48.453552 29292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3397a014bc5747e8964b53d3291d0d9f" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37491 } }
I20260926 20:10:48.453974 29292 leader_election.cc:302] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3397a014bc5747e8964b53d3291d0d9f; no voters:
I20260926 20:10:48.454293 29292 leader_election.cc:288] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:48.454526 29297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:48.454768 29297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [term 1 LEADER]: Becoming Leader. State: Replica: 3397a014bc5747e8964b53d3291d0d9f, State: Running, Role: LEADER
I20260926 20:10:48.455076 29297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3397a014bc5747e8964b53d3291d0d9f" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37491 } }
I20260926 20:10:48.455648 29292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:48.456841 29298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3397a014bc5747e8964b53d3291d0d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3397a014bc5747e8964b53d3291d0d9f" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37491 } } }
I20260926 20:10:48.456988 29298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:48.457259 29299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3397a014bc5747e8964b53d3291d0d9f. Latest consensus state: current_term: 1 leader_uuid: "3397a014bc5747e8964b53d3291d0d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3397a014bc5747e8964b53d3291d0d9f" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37491 } } }
I20260926 20:10:48.457365 29299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:48.457459 29306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:48.460703 29306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:48.466506 29306 catalog_manager.cc:1347] Generated new cluster ID: 55a13b403aca42319667d82374aeae91
I20260926 20:10:48.466594 29306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:48.484910 29306 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:48.486112 29306 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:48.495573 29306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3397a014bc5747e8964b53d3291d0d9f: Generated new TSK 0
I20260926 20:10:48.496474 29306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:10:48.574268 29295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:48.517133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.23.50.190:37491)
0926 20:10:48.517288 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:48.517293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:48.517497 (+ 204us) server_negotiation.cc:407] Connection header received
0926 20:10:48.517650 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:48.517653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:48.517713 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:48.517840 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0926 20:10:48.518187 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:48.519000 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:48.519940 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:48.520290 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:48.520388 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:48.520711 (+ 323us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 20:10:48.573690 (+ 52979us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:10:48.574094 (+ 404us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55840: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 20:10:48.574362 29295 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55840: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 20:10:48.574394 29321 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:10:48.517032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37491 (local address 127.0.0.1:55840)
0926 20:10:48.517406 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:48.517419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:48.517524 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:48.517939 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:48.517942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:48.517950 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 20:10:48.518059 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:48.518065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:48.519132 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:48.519134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:48.519823 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:48.519832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:48.520425 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:48.520428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:48.520500 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:48.520526 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 20:10:48.573990 (+ 53464us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 20:10:48.574205 (+ 215us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.50.190:37491: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":127,"threads_started":1}
I20260926 20:10:48.575263 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 29225
2026-09-26T20:10:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (349 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (127 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:10:48 dist-test-slave-7034 krb5kdc[29333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:10:48 dist-test-slave-7034 krb5kdc[29333](info): set up 2 sockets
Sep 26 20:10:48 dist-test-slave-7034 krb5kdc[29333](info): commencing operation
krb5kdc: starting...
W20260926 20:10:49.878722 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:10:49 dist-test-slave-7034 krb5kdc[29333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:10:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:10:50.005321 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37111
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:37111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:50.118692 29346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:50.119036 29346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:50.119112 29346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:50.123308 29346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:50.123400 29346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:50.123438 29346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:50.123498 29346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:50.123539 29346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:50.129187 29346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:37111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:50.130463 29346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:50.131524 29346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:10:50.140599 29346 server_base.cc:1034] running on GCE node
W20260926 20:10:50.140583 29352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:50.140797 29354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:50.140583 29351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:50.141363 29346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:50.142601 29346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:50.143780 29346 hybrid_clock.cc:648] HybridClock initialized: now 1790453450143763 us; error 29 us; skew 500 ppm
Sep 26 20:10:50 dist-test-slave-7034 krb5kdc[29333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:50.146647 29346 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:10:50.147552 29346 webserver.cc:460] Webserver started at http://127.23.50.190:45081/ using document root <none> and password file <none>
I20260926 20:10:50.148150 29346 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:50.148244 29346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:50.148484 29346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:50.150732 29346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "9bfacd4a0d244c71a10e7a13e7d78f1c"
format_stamp: "Formatted at 2026-09-26 20:10:50 on dist-test-slave-7034"
server_key: "e89a5e648df167388f0237acdf13e65d"
server_key_iv: "981c0ae6ab475f4826c0117478da502b"
server_key_version: "encryptionkey@0"
I20260926 20:10:50.151345 29346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "9bfacd4a0d244c71a10e7a13e7d78f1c"
format_stamp: "Formatted at 2026-09-26 20:10:50 on dist-test-slave-7034"
server_key: "e89a5e648df167388f0237acdf13e65d"
server_key_iv: "981c0ae6ab475f4826c0117478da502b"
server_key_version: "encryptionkey@0"
I20260926 20:10:50.155272 29346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:10:50.158073 29361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:50.159505 29346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:50.159664 29346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "9bfacd4a0d244c71a10e7a13e7d78f1c"
format_stamp: "Formatted at 2026-09-26 20:10:50 on dist-test-slave-7034"
server_key: "e89a5e648df167388f0237acdf13e65d"
server_key_iv: "981c0ae6ab475f4826c0117478da502b"
server_key_version: "encryptionkey@0"
I20260926 20:10:50.159797 29346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:50.170821 29346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:50.173933 29346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:50.174142 29346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:50.181417 29346 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:37111
I20260926 20:10:50.181483 29413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:37111 every 8 connection(s)
I20260926 20:10:50.182521 29346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:50.185398 29414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:50.190997 29414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c: Bootstrap starting.
I20260926 20:10:50.191877 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 29346
I20260926 20:10:50.191983 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:50.192279 23754 external_mini_cluster.cc:1428] Setting key c2b0744ea7db4d12a5281d86f539cc77
I20260926 20:10:50.193931 29414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:50.194852 29414 log.cc:824] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:50.197209 29414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c: No bootstrap required, opened a new log
I20260926 20:10:50.201339 29414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfacd4a0d244c71a10e7a13e7d78f1c" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37111 } }
I20260926 20:10:50.201578 29414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:50.201620 29414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfacd4a0d244c71a10e7a13e7d78f1c, State: Initialized, Role: FOLLOWER
I20260926 20:10:50.202085 29414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfacd4a0d244c71a10e7a13e7d78f1c" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37111 } }
I20260926 20:10:50.202217 29414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:50.202260 29414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:50.202335 29414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:50.203226 29414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfacd4a0d244c71a10e7a13e7d78f1c" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37111 } }
I20260926 20:10:50.203547 29414 leader_election.cc:302] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bfacd4a0d244c71a10e7a13e7d78f1c; no voters:
I20260926 20:10:50.203825 29414 leader_election.cc:288] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:50.203989 29419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:50.204288 29419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [term 1 LEADER]: Becoming Leader. State: Replica: 9bfacd4a0d244c71a10e7a13e7d78f1c, State: Running, Role: LEADER
I20260926 20:10:50.204612 29419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfacd4a0d244c71a10e7a13e7d78f1c" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37111 } }
I20260926 20:10:50.205180 29414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:50.206496 29420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bfacd4a0d244c71a10e7a13e7d78f1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfacd4a0d244c71a10e7a13e7d78f1c" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37111 } } }
I20260926 20:10:50.206650 29420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:50.206959 29421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bfacd4a0d244c71a10e7a13e7d78f1c. Latest consensus state: current_term: 1 leader_uuid: "9bfacd4a0d244c71a10e7a13e7d78f1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfacd4a0d244c71a10e7a13e7d78f1c" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37111 } } }
I20260926 20:10:50.207037 29421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c [sys.catalog]: This master's current role is: LEADER
Sep 26 20:10:50 dist-test-slave-7034 krb5kdc[29333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
W20260926 20:10:50.211208 29434 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:10:50.211308 29434 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:10:50.211377 29428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:50.212244 29428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:50.215296 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:50.193489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33912 (local address 127.23.50.190:37111)
0926 20:10:50.194009 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:50.194029 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:50.194073 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:10:50.194959 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:50.194984 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:50.195612 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:50.196048 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:50.196725 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.198202 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.199151 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.199938 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.200143 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:50.209110 (+ 8967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:50.209118 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:50.209136 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:50.209185 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:50.211490 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:50.212140 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:50.212148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:50.212154 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:50.212320 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:50.212706 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:50.212710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:50.212712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:50.213269 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:50.213463 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:50.213822 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 20:10:50.214195 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":130,"threads_started":1}
I20260926 20:10:50.218375 29428 catalog_manager.cc:1347] Generated new cluster ID: f7b81e40a7d94799aa1d853e6782706e
I20260926 20:10:50.218473 29428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:50.226725 29428 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:50.227962 29428 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:50.232961 29428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bfacd4a0d244c71a10e7a13e7d78f1c: Generated new TSK 0
I20260926 20:10:50.233956 29428 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:10:50.269639 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37111
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:50.385356 29440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:50.385700 29440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:50.385787 29440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:50.390118 29440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:50.390237 29440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:50.390341 29440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:10:50.395979 29440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:50.397188 29440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:50.398344 29440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:50.408484 29445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:50.408489 29446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:50.408550 29440 server_base.cc:1034] running on GCE node
W20260926 20:10:50.408489 29448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:50.409283 29440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:50.410099 29440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:50.411296 29440 hybrid_clock.cc:648] HybridClock initialized: now 1790453450411284 us; error 31 us; skew 500 ppm
Sep 26 20:10:50 dist-test-slave-7034 krb5kdc[29333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:50.414551 29440 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:10:50.415587 29440 webserver.cc:460] Webserver started at http://127.23.50.129:44827/ using document root <none> and password file <none>
I20260926 20:10:50.416239 29440 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:50.416304 29440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:50.416553 29440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:50.418768 29440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "eb7cb3514c8449768dbed273ace685d2"
format_stamp: "Formatted at 2026-09-26 20:10:50 on dist-test-slave-7034"
server_key: "209db314a32e6f64ebf20b9e7e44dc6a"
server_key_iv: "b8d8805b45704121bc4d5870c51d0429"
server_key_version: "encryptionkey@0"
I20260926 20:10:50.419335 29440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "eb7cb3514c8449768dbed273ace685d2"
format_stamp: "Formatted at 2026-09-26 20:10:50 on dist-test-slave-7034"
server_key: "209db314a32e6f64ebf20b9e7e44dc6a"
server_key_iv: "b8d8805b45704121bc4d5870c51d0429"
server_key_version: "encryptionkey@0"
I20260926 20:10:50.423237 29440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:10:50.426056 29455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:50.427625 29440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:50.427799 29440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "eb7cb3514c8449768dbed273ace685d2"
format_stamp: "Formatted at 2026-09-26 20:10:50 on dist-test-slave-7034"
server_key: "209db314a32e6f64ebf20b9e7e44dc6a"
server_key_iv: "b8d8805b45704121bc4d5870c51d0429"
server_key_version: "encryptionkey@0"
I20260926 20:10:50.427951 29440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:50.464043 29440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:50.467844 29440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:50.468070 29440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:50.468740 29440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:50.469655 29440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:50.469731 29440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:50.469811 29440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:50.469897 29440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 20:10:50.479468 29440 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:42439
I20260926 20:10:50.479486 29568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:42439 every 8 connection(s)
I20260926 20:10:50.480468 29440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:10:50.487385 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 29440
I20260926 20:10:50.487550 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:10:50.487825 23754 external_mini_cluster.cc:1428] Setting key 0ab7993e8904454ec1d821b4546ef640
Sep 26 20:10:50 dist-test-slave-7034 krb5kdc[29333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:50.496200 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:50.482460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:56789 (local address 127.23.50.190:37111)
0926 20:10:50.482604 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:50.482608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:50.483548 (+ 940us) server_negotiation.cc:407] Connection header received
0926 20:10:50.484961 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:50.484964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:50.485027 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:50.485138 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:50.486500 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.487048 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.488508 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.488815 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.488921 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:50.491984 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:50.491990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:50.491996 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:50.492030 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:50.493868 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:50.494462 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:50.494469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:50.494474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:50.494574 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:50.494919 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:50.494925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:50.494929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:50.495212 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:50.495334 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:50.495846 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 20:10:50.495988 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:10:50.496980 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:50.482861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37111 (local address 127.23.50.129:56789)
0926 20:10:50.483374 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:50.483422 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:50.484718 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:50.485333 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:50.485341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:50.485722 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:50.486343 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:50.486359 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.487198 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.487201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:50.488342 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:50.488354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.488972 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.488975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:50.489230 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:50.489909 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:50.489931 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:50.491791 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:50.494015 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:50.494020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:50.494034 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:50.494349 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:50.494677 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:50.494679 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:50.494680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:50.494821 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:50.495324 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:50.495329 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:50.495588 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 20:10:50.495754 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 20:10:50.496020 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":135,"threads_started":1}
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:10:50.498411 29569 heartbeater.cc:347] Connected to a master server at 127.23.50.190:37111
I20260926 20:10:50.498677 29569 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:50.499159 29569 heartbeater.cc:511] Master 127.23.50.190:37111 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
I20260926 20:10:50.500900 29378 ts_manager.cc:194] Registered new tserver with Master: eb7cb3514c8449768dbed273ace685d2 (127.23.50.129:42439)
I20260926 20:10:50.502158 29378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:56789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:10:50.517939 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37111
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:50.632721 29574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:50.633044 29574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:50.633117 29574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:50.637228 29574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:50.637341 29574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:50.637439 29574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:10:50.642864 29574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:50.644042 29574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:50.645136 29574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:10:50.655596 29574 server_base.cc:1034] running on GCE node
W20260926 20:10:50.655619 29582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:50.655781 29580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:50.655619 29579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:50.656283 29574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:50.656930 29574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:50.658095 29574 hybrid_clock.cc:648] HybridClock initialized: now 1790453450658056 us; error 61 us; skew 500 ppm
Sep 26 20:10:50 dist-test-slave-7034 krb5kdc[29333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:50.660993 29574 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:10:50.662041 29574 webserver.cc:460] Webserver started at http://127.23.50.130:37703/ using document root <none> and password file <none>
I20260926 20:10:50.662628 29574 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:50.662724 29574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:50.662972 29574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:50.665206 29574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "645fd6d399244932bb9655525dbb3469"
format_stamp: "Formatted at 2026-09-26 20:10:50 on dist-test-slave-7034"
server_key: "80f4f3a43a0666bf37d0403cbc8121bc"
server_key_iv: "0b531fc46e318f62d5dab650c9875260"
server_key_version: "encryptionkey@0"
I20260926 20:10:50.665796 29574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "645fd6d399244932bb9655525dbb3469"
format_stamp: "Formatted at 2026-09-26 20:10:50 on dist-test-slave-7034"
server_key: "80f4f3a43a0666bf37d0403cbc8121bc"
server_key_iv: "0b531fc46e318f62d5dab650c9875260"
server_key_version: "encryptionkey@0"
I20260926 20:10:50.669636 29574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260926 20:10:50.672240 29589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:50.673561 29574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:10:50.673717 29574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "645fd6d399244932bb9655525dbb3469"
format_stamp: "Formatted at 2026-09-26 20:10:50 on dist-test-slave-7034"
server_key: "80f4f3a43a0666bf37d0403cbc8121bc"
server_key_iv: "0b531fc46e318f62d5dab650c9875260"
server_key_version: "encryptionkey@0"
I20260926 20:10:50.673840 29574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:50.699366 29574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:50.702853 29574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:50.703068 29574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:50.703692 29574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:50.704605 29574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:50.704671 29574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:50.704754 29574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:50.704808 29574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:50.714082 29574 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:34879
I20260926 20:10:50.714100 29702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:34879 every 8 connection(s)
I20260926 20:10:50.715104 29574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:10:50.724977 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 29574
I20260926 20:10:50.725109 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
Sep 26 20:10:50 dist-test-slave-7034 krb5kdc[29333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:50.725422 23754 external_mini_cluster.cc:1428] Setting key aaded98e102c4c951dfa6a1696ab0b96
I20260926 20:10:50.730324 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:50.717017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:60423 (local address 127.23.50.190:37111)
0926 20:10:50.717225 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:50.717230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:50.718123 (+ 893us) server_negotiation.cc:407] Connection header received
0926 20:10:50.719575 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:50.719578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:50.719630 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:50.719710 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:50.721098 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.721663 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.723004 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.723237 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.723340 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:50.726087 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:50.726091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:50.726094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:50.726126 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:50.727951 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:50.728595 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:50.728602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:50.728607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:50.728709 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:50.729112 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:50.729116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:50.729118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:50.729326 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:50.729439 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:50.730029 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 20:10:50.730158 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:10:50.731237 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:50.717387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37111 (local address 127.23.50.130:60423)
0926 20:10:50.717949 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:50.718002 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:50.719340 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:50.719908 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:50.719915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:50.720352 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:50.720895 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:50.720910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.721810 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.721813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:50.722870 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:50.722880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.723391 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.723394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:50.723548 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:50.724224 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:50.724242 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:50.725844 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:50.728082 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:50.728088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:50.728119 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:50.728464 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:50.728802 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:50.728805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:50.728806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:50.728998 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:50.729451 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:50.729460 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:50.729734 (+ 274us) client_negotiation.cc:769] Sending connection context
0926 20:10:50.729974 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 20:10:50.730368 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":118,"threads_started":1}
I20260926 20:10:50.732563 29703 heartbeater.cc:347] Connected to a master server at 127.23.50.190:37111
I20260926 20:10:50.732877 29703 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:50.733410 29703 heartbeater.cc:511] Master 127.23.50.190:37111 requested a full tablet report, sending...
I20260926 20:10:50.734634 29378 ts_manager.cc:194] Registered new tserver with Master: 645fd6d399244932bb9655525dbb3469 (127.23.50.130:34879)
I20260926 20:10:50.735301 29378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:60423
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:10:50.757745 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37111
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:50.873092 29708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:50.873377 29708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:50.873420 29708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:50.877655 29708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:50.877725 29708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:50.877785 29708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:10:50.883572 29708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:50.884688 29708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:50.885689 29708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:50.895984 29713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:50.896086 29708 server_base.cc:1034] running on GCE node
W20260926 20:10:50.895955 29716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:50.895955 29714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:50.896632 29708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:50.897305 29708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:50.898491 29708 hybrid_clock.cc:648] HybridClock initialized: now 1790453450898453 us; error 49 us; skew 500 ppm
Sep 26 20:10:50 dist-test-slave-7034 krb5kdc[29333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:50.901451 29708 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:10:50.902392 29708 webserver.cc:460] Webserver started at http://127.23.50.131:41685/ using document root <none> and password file <none>
I20260926 20:10:50.902918 29708 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:50.902978 29708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:50.903152 29708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:50.905285 29708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "baa9f01c457242fd99db42bf9c20c841"
format_stamp: "Formatted at 2026-09-26 20:10:50 on dist-test-slave-7034"
server_key: "e1f281facec0d798d981f27acae9bcbe"
server_key_iv: "828fe4f6415420ec7e31a98fcd183bea"
server_key_version: "encryptionkey@0"
I20260926 20:10:50.905953 29708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "baa9f01c457242fd99db42bf9c20c841"
format_stamp: "Formatted at 2026-09-26 20:10:50 on dist-test-slave-7034"
server_key: "e1f281facec0d798d981f27acae9bcbe"
server_key_iv: "828fe4f6415420ec7e31a98fcd183bea"
server_key_version: "encryptionkey@0"
I20260926 20:10:50.909808 29708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:10:50.912540 29723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:50.914052 29708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:50.914183 29708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "baa9f01c457242fd99db42bf9c20c841"
format_stamp: "Formatted at 2026-09-26 20:10:50 on dist-test-slave-7034"
server_key: "e1f281facec0d798d981f27acae9bcbe"
server_key_iv: "828fe4f6415420ec7e31a98fcd183bea"
server_key_version: "encryptionkey@0"
I20260926 20:10:50.914290 29708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:50.927925 29708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:50.931895 29708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:50.932076 29708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:50.932780 29708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:50.933676 29708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:50.933727 29708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:50.933772 29708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:50.933790 29708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:50.943995 29708 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:40529
I20260926 20:10:50.944015 29836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:40529 every 8 connection(s)
I20260926 20:10:50.945055 29708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:10:50.954964 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 29708
I20260926 20:10:50.955111 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:10:50.955397 23754 external_mini_cluster.cc:1428] Setting key cbd8abd0e4eafdb2f3abd850e0c39694
Sep 26 20:10:50 dist-test-slave-7034 krb5kdc[29333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:10:50.960793 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:50.947290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:51267 (local address 127.23.50.190:37111)
0926 20:10:50.947452 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:50.947455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:50.948410 (+ 955us) server_negotiation.cc:407] Connection header received
0926 20:10:50.950108 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:50.950111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:50.950164 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:50.950269 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:50.951633 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.952165 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.953431 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.953647 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.953734 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:50.956895 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:50.956899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:50.956903 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:50.956934 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:50.958699 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:50.959268 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:50.959270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:50.959272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:50.959354 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:50.959753 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:50.959755 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:50.959757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:50.959947 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:50.960022 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:50.960553 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 20:10:50.960656 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:10:50.961793 29839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:50.947675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37111 (local address 127.23.50.131:51267)
0926 20:10:50.948234 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:50.948283 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:50.949799 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:50.950465 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:50.950473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:50.950919 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:50.951458 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:50.951472 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.952298 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.952301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:50.953308 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:50.953317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.953822 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.953826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:50.954060 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:50.954724 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:50.954742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:50.956676 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:50.958843 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:50.958850 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:50.958864 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:50.959171 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:50.959468 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:50.959471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:50.959472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:50.959637 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:50.960038 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:50.960043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:50.960328 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 20:10:50.960501 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 20:10:50.960907 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":122,"threads_started":1}
I20260926 20:10:50.963024 29837 heartbeater.cc:347] Connected to a master server at 127.23.50.190:37111
I20260926 20:10:50.963292 29837 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:50.963852 29837 heartbeater.cc:511] Master 127.23.50.190:37111 requested a full tablet report, sending...
I20260926 20:10:50.964998 29378 ts_manager.cc:194] Registered new tserver with Master: baa9f01c457242fd99db42bf9c20c841 (127.23.50.131:40529)
I20260926 20:10:50.965605 29378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:51267
I20260926 20:10:50.970085 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/test-admin.keytab.
Sep 26 20:10:50 dist-test-slave-7034 krb5kdc[29333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:50.987625 23754 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 20:10:50.990641 29844 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:10:50.989117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42382 (local address 127.23.50.129:42439)
0926 20:10:50.989422 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:50.989439 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:50.989468 (+ 29us) server_negotiation.cc:407] Connection header received
0926 20:10:50.989535 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:50.989540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:50.989728 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:50.989921 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:50.990425 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42382: BlockingRecv error: recv got EOF from 127.0.0.1:42382 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":167,"threads_started":1}
W20260926 20:10:50.990972 23754 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:10:50 dist-test-slave-7034 krb5kdc[29333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:50.992079 23754 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:10:50 dist-test-slave-7034 krb5kdc[29333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:50.993036 23754 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:10:50 dist-test-slave-7034 krb5kdc[29333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.129@KRBTEST.COM
I20260926 20:10:51.003476 29844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:50.994090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.23.50.129:42439)
0926 20:10:50.994234 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:50.994240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:50.994365 (+ 125us) server_negotiation.cc:407] Connection header received
0926 20:10:50.994539 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:50.994543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:50.994595 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:50.994680 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:50.995245 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.996053 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.996927 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:50.997334 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:50.997456 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:50.999901 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:50.999917 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:50.999932 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:50.999958 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:51.001530 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:51.002097 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:51.002104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:51.002106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:51.002191 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:51.002477 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:51.002480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:51.002482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:51.002750 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:51.002997 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:51.003168 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 20:10:51.003317 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 20:10:51.007905 29844 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:10:51.006768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.23.50.129:42439)
0926 20:10:51.006915 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:51.006919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:51.007015 (+ 96us) server_negotiation.cc:407] Connection header received
0926 20:10:51.007182 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:51.007185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:51.007233 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:51.007315 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:51.007787 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42406: BlockingRecv error: recv got EOF from 127.0.0.1:42406 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 20:10:51.008064 23754 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:10:51 dist-test-slave-7034 krb5kdc[29333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:51.009249 23754 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:10:51 dist-test-slave-7034 krb5kdc[29333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:51.010385 23754 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:10:51 dist-test-slave-7034 krb5kdc[29333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.129@KRBTEST.COM
I20260926 20:10:51.019850 29844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:51.011281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.23.50.129:42439)
0926 20:10:51.011407 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:51.011412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:51.011566 (+ 154us) server_negotiation.cc:407] Connection header received
0926 20:10:51.011735 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:51.011743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:51.011783 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:51.011866 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:51.012364 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:51.012876 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:51.013673 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:51.013966 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:51.014077 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:51.016456 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:51.016465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:51.016467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:51.016493 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:51.018137 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:51.018628 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:51.018632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:51.018633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:51.018732 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:51.019017 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:51.019020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:51.019022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:51.019216 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:51.019334 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:51.019588 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 20:10:51.019724 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 20:10:51.022955 29844 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:10:51.021860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.23.50.129:42439)
0926 20:10:51.021992 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:51.021997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:51.022129 (+ 132us) server_negotiation.cc:407] Connection header received
0926 20:10:51.022287 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:51.022290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:51.022336 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:51.022420 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:51.022833 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42424: BlockingRecv error: recv got EOF from 127.0.0.1:42424 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 20:10:51.023103 23754 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790453421225356-23754-0/krb5kdc/krb5cc' of type FILE
Sep 26 20:10:51 dist-test-slave-7034 krb5kdc[29333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:51.024156 23754 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 20:10:51 dist-test-slave-7034 krb5kdc[29333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:51.025049 23754 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 20:10:51 dist-test-slave-7034 krb5kdc[29333](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.129@KRBTEST.COM
I20260926 20:10:51.034734 29844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:51.025902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42438 (local address 127.23.50.129:42439)
0926 20:10:51.026069 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:51.026074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:51.026107 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:10:51.026248 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:51.026250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:51.026290 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:51.026363 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:51.026829 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:51.027380 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:51.028338 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:51.028578 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:51.028675 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:51.031381 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:51.031384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:51.031387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:51.031414 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:51.033000 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:51.033510 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:51.033513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:51.033514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:51.033598 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:51.033916 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:51.033918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:51.033919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:51.034090 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:51.034206 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:51.034408 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 20:10:51.034550 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:10:51.035821 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 29440
I20260926 20:10:51.041270 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 29574
I20260926 20:10:51.046660 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 29708
I20260926 20:10:51.052217 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 29346
2026-09-26T20:10:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2224 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:10:51 dist-test-slave-7034 krb5kdc[29857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:10:51 dist-test-slave-7034 krb5kdc[29857](info): set up 2 sockets
Sep 26 20:10:51 dist-test-slave-7034 krb5kdc[29857](info): commencing operation
krb5kdc: starting...
W20260926 20:10:52.135076 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:10:52 dist-test-slave-7034 krb5kdc[29857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.190.keytab.
Entry for principal oryx/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.190.keytab.
I20260926 20:10:52.264989 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:41147
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.190.keytab
--principal=oryx/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:41147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:52.379428 29870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:52.379761 29870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:52.379841 29870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:52.383914 29870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:52.384013 29870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:52.384052 29870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:52.384105 29870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:52.384153 29870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:52.389611 29870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:41147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.190.keytab
--principal=oryx/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:41147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:52.390831 29870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:52.391989 29870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:52.400761 29875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:52.400763 29876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:52.400789 29878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:52.401073 29870 server_base.cc:1034] running on GCE node
I20260926 20:10:52.401649 29870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:52.402879 29870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:52.404081 29870 hybrid_clock.cc:648] HybridClock initialized: now 1790453452404050 us; error 53 us; skew 500 ppm
Sep 26 20:10:52 dist-test-slave-7034 krb5kdc[29857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453452, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:52.407155 29870 init.cc:377] Logged in from keytab as oryx/127.23.50.190@KRBTEST.COM (short username oryx)
I20260926 20:10:52.408156 29870 webserver.cc:460] Webserver started at http://127.23.50.190:39315/ using document root <none> and password file <none>
I20260926 20:10:52.408752 29870 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:52.408849 29870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:52.409088 29870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:52.411322 29870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "8c0a1ed2d1614a04add0c82a6c22d24e"
format_stamp: "Formatted at 2026-09-26 20:10:52 on dist-test-slave-7034"
server_key: "b5b6b6adad2c53dcd3b81c20418e0e5b"
server_key_iv: "7c616a9771c56ab8059de26381ec3798"
server_key_version: "encryptionkey@0"
I20260926 20:10:52.411933 29870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "8c0a1ed2d1614a04add0c82a6c22d24e"
format_stamp: "Formatted at 2026-09-26 20:10:52 on dist-test-slave-7034"
server_key: "b5b6b6adad2c53dcd3b81c20418e0e5b"
server_key_iv: "7c616a9771c56ab8059de26381ec3798"
server_key_version: "encryptionkey@0"
I20260926 20:10:52.416049 29870 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:10:52.418840 29885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:52.420302 29870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 20:10:52.420461 29870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "8c0a1ed2d1614a04add0c82a6c22d24e"
format_stamp: "Formatted at 2026-09-26 20:10:52 on dist-test-slave-7034"
server_key: "b5b6b6adad2c53dcd3b81c20418e0e5b"
server_key_iv: "7c616a9771c56ab8059de26381ec3798"
server_key_version: "encryptionkey@0"
I20260926 20:10:52.420588 29870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:52.432498 29870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:52.435873 29870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:52.436106 29870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:52.443674 29870 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:41147
I20260926 20:10:52.444046 29937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:41147 every 8 connection(s)
I20260926 20:10:52.444747 29870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:52.447674 29938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:52.451956 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 29870
I20260926 20:10:52.452090 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:52.452386 23754 external_mini_cluster.cc:1428] Setting key 9f9c9c87870679f6f992360a6ba42471
I20260926 20:10:52.453711 29938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e: Bootstrap starting.
I20260926 20:10:52.456116 29938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:52.456904 29938 log.cc:824] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:52.458886 29938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e: No bootstrap required, opened a new log
Sep 26 20:10:52 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
I20260926 20:10:52.463124 29938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } }
I20260926 20:10:52.463419 29938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:52.463503 29938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0a1ed2d1614a04add0c82a6c22d24e, State: Initialized, Role: FOLLOWER
I20260926 20:10:52.463981 29938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } }
I20260926 20:10:52.464102 29938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:52.464191 29938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:52.464341 29938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:52.465256 29938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } }
I20260926 20:10:52.465600 29938 leader_election.cc:302] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c0a1ed2d1614a04add0c82a6c22d24e; no voters:
I20260926 20:10:52.465998 29938 leader_election.cc:288] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:52.466075 29943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:52.466334 29943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 1 LEADER]: Becoming Leader. State: Replica: 8c0a1ed2d1614a04add0c82a6c22d24e, State: Running, Role: LEADER
I20260926 20:10:52.466706 29943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } }
I20260926 20:10:52.467360 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:52.453875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.23.50.190:41147)
0926 20:10:52.454398 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:52.454415 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:52.454454 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:10:52.455141 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:52.455159 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:52.455509 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:52.455885 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:52.456715 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:52.457756 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:52.458640 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:52.459188 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:52.459369 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:52.462232 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:52.462237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:52.462252 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:52.462282 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:52.464106 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:52.464597 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:52.464610 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:52.464615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:52.464727 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:52.465034 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:52.465037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:52.465039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:52.465506 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:52.465709 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:52.466015 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 20:10:52.466352 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":150,"threads_started":1}
I20260926 20:10:52.467478 29938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:52.468047 29943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } }
I20260926 20:10:52.468182 29943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:52.468549 29944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c0a1ed2d1614a04add0c82a6c22d24e. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } }
I20260926 20:10:52.468703 29944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: This master's current role is: LEADER
W20260926 20:10:52.472361 29958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:10:52.472433 29958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:10:52.472520 29947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:52.473212 29947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:52.478670 29947 catalog_manager.cc:1347] Generated new cluster ID: d249234698a84df49b235ad8f79ffee2
I20260926 20:10:52.478752 29947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:52.504185 29947 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:52.505255 29947 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:52.512993 29947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e: Generated new TSK 0
I20260926 20:10:52.513777 29947 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab.
Entry for principal oryx/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab.
I20260926 20:10:52.549633 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:41147
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab
--principal=oryx/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:52.665805 29964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:52.666149 29964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:52.666239 29964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:52.670877 29964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:52.670982 29964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:52.671082 29964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:10:52.677030 29964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab
--principal=oryx/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:41147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:52.678277 29964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:52.679335 29964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:52.689785 29972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:52.689921 29964 server_base.cc:1034] running on GCE node
W20260926 20:10:52.689786 29970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:52.689785 29969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:52.690575 29964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:52.691270 29964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:52.692468 29964 hybrid_clock.cc:648] HybridClock initialized: now 1790453452692414 us; error 67 us; skew 500 ppm
Sep 26 20:10:52 dist-test-slave-7034 krb5kdc[29857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453452, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:52.695647 29964 init.cc:377] Logged in from keytab as oryx/127.23.50.129@KRBTEST.COM (short username oryx)
I20260926 20:10:52.696724 29964 webserver.cc:460] Webserver started at http://127.23.50.129:41235/ using document root <none> and password file <none>
I20260926 20:10:52.697367 29964 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:52.697433 29964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:52.697664 29964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:52.699946 29964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "0c3b0b17121f483789d7ea584b9a5a79"
format_stamp: "Formatted at 2026-09-26 20:10:52 on dist-test-slave-7034"
server_key: "1255aab8bdecabdb518c2785befbfec7"
server_key_iv: "d0834b78491e180a8517977ced3483a5"
server_key_version: "encryptionkey@0"
I20260926 20:10:52.700574 29964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "0c3b0b17121f483789d7ea584b9a5a79"
format_stamp: "Formatted at 2026-09-26 20:10:52 on dist-test-slave-7034"
server_key: "1255aab8bdecabdb518c2785befbfec7"
server_key_iv: "d0834b78491e180a8517977ced3483a5"
server_key_version: "encryptionkey@0"
I20260926 20:10:52.704783 29964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 20:10:52.707703 29979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:52.709088 29964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:10:52.709227 29964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "0c3b0b17121f483789d7ea584b9a5a79"
format_stamp: "Formatted at 2026-09-26 20:10:52 on dist-test-slave-7034"
server_key: "1255aab8bdecabdb518c2785befbfec7"
server_key_iv: "d0834b78491e180a8517977ced3483a5"
server_key_version: "encryptionkey@0"
I20260926 20:10:52.709358 29964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:52.723268 29964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:52.726905 29964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:52.727131 29964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:52.727794 29964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:52.728727 29964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:52.728783 29964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:52.728856 29964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:52.728889 29964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:52.739051 29964 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:36633
I20260926 20:10:52.739095 30092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:36633 every 8 connection(s)
I20260926 20:10:52.740114 29964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:10:52.747201 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 29964
I20260926 20:10:52.747371 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:10:52.747684 23754 external_mini_cluster.cc:1428] Setting key 387f809297c681f17ba60daf94d1d4ed
Sep 26 20:10:52 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453452, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
I20260926 20:10:52.758044 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:52.742297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:51927 (local address 127.23.50.190:41147)
0926 20:10:52.742462 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:52.742467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:52.743555 (+ 1088us) server_negotiation.cc:407] Connection header received
0926 20:10:52.745156 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:52.745160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:52.745216 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:52.745301 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:52.747034 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:52.747748 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:52.749602 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:52.749911 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:52.750026 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:52.753581 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:52.753584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:52.753587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:52.753618 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:52.755290 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:52.756071 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:52.756073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:52.756075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:52.756169 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:52.756635 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:52.756639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:52.756641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:52.756850 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:52.756944 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:52.757723 (+ 779us) server_negotiation.cc:299] Negotiation successful
0926 20:10:52.757905 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for oryx/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:10:52.759243 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:52.742817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.129:51927)
0926 20:10:52.743367 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:52.743418 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:52.744887 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:52.745539 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:52.745548 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:52.746199 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:52.746780 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:52.746798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:52.747949 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:52.747953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:52.749440 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:52.749474 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:52.750109 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:52.750117 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:52.750407 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:52.751222 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:52.751251 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:52.753316 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:52.755452 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:52.755463 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:52.755488 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:52.755935 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:52.756282 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:52.756287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:52.756289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:52.756509 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:52.756975 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:52.756984 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:52.757395 (+ 411us) client_negotiation.cc:769] Sending connection context
0926 20:10:52.757667 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 20:10:52.758150 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":150,"threads_started":1}
I20260926 20:10:52.760878 30093 heartbeater.cc:347] Connected to a master server at 127.23.50.190:41147
I20260926 20:10:52.761215 30093 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:52.761940 30093 heartbeater.cc:511] Master 127.23.50.190:41147 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.130@KRBTEST.COM" created.
I20260926 20:10:52.764017 29902 ts_manager.cc:194] Registered new tserver with Master: 0c3b0b17121f483789d7ea584b9a5a79 (127.23.50.129:36633)
I20260926 20:10:52.765252 29902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:51927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab.
Entry for principal oryx/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab.
I20260926 20:10:52.782873 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:41147
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab
--principal=oryx/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:52.900123 30098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:52.900404 30098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:52.900480 30098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:52.904639 30098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:52.904702 30098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:52.904794 30098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:10:52.910270 30098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab
--principal=oryx/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:41147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:52.911372 30098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:52.912412 30098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:52.922288 30104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:52.922405 30103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:52.922288 30106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:52.922590 30098 server_base.cc:1034] running on GCE node
I20260926 20:10:52.923044 30098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:52.923790 30098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:52.925019 30098 hybrid_clock.cc:648] HybridClock initialized: now 1790453452924998 us; error 34 us; skew 500 ppm
Sep 26 20:10:52 dist-test-slave-7034 krb5kdc[29857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453452, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:52.928462 30098 init.cc:377] Logged in from keytab as oryx/127.23.50.130@KRBTEST.COM (short username oryx)
I20260926 20:10:52.929543 30098 webserver.cc:460] Webserver started at http://127.23.50.130:34051/ using document root <none> and password file <none>
I20260926 20:10:52.930254 30098 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:52.930352 30098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:52.930725 30098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:52.933143 30098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "c7758e9f4ea346a394c342bea4de2728"
format_stamp: "Formatted at 2026-09-26 20:10:52 on dist-test-slave-7034"
server_key: "349a457fbd0069d4b61a40390f5e2a66"
server_key_iv: "ad3919af6d70294998d85ec5059d3af1"
server_key_version: "encryptionkey@0"
I20260926 20:10:52.933756 30098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "c7758e9f4ea346a394c342bea4de2728"
format_stamp: "Formatted at 2026-09-26 20:10:52 on dist-test-slave-7034"
server_key: "349a457fbd0069d4b61a40390f5e2a66"
server_key_iv: "ad3919af6d70294998d85ec5059d3af1"
server_key_version: "encryptionkey@0"
I20260926 20:10:52.937693 30098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:10:52.940388 30113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:52.942019 30098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:52.942166 30098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "c7758e9f4ea346a394c342bea4de2728"
format_stamp: "Formatted at 2026-09-26 20:10:52 on dist-test-slave-7034"
server_key: "349a457fbd0069d4b61a40390f5e2a66"
server_key_iv: "ad3919af6d70294998d85ec5059d3af1"
server_key_version: "encryptionkey@0"
I20260926 20:10:52.942296 30098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:52.978643 30098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:52.982379 30098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:52.982602 30098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:52.983245 30098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:52.984115 30098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:52.984196 30098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:52.984273 30098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:52.984310 30098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:52.995802 30098 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:37165
I20260926 20:10:52.995880 30226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:37165 every 8 connection(s)
I20260926 20:10:52.996969 30098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:10:53.000519 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 30098
I20260926 20:10:53.000653 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:10:53.001026 23754 external_mini_cluster.cc:1428] Setting key 1eb06f55972a43fe9c306a132574004c
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453452, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:10:53.013396 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:52.999161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:50109 (local address 127.23.50.190:41147)
0926 20:10:52.999305 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:52.999308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.000231 (+ 923us) server_negotiation.cc:407] Connection header received
0926 20:10:53.002056 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.002059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.002104 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.002170 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.003819 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.004440 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.005742 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.006049 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.006177 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.009189 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.009193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.009195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.009222 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.010831 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.011491 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.011494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.011496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.011598 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.011989 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.011999 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.012000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.012227 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.012363 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.012945 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.013133 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.131@KRBTEST.COM" created.
I20260926 20:10:53.014514 30229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:52.999608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.130:50109)
0926 20:10:53.000070 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.000114 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.001797 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.002368 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.002383 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.002887 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.003638 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.003661 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.004584 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.004587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.005522 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.005532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.006288 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.006293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.006460 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.007108 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.007125 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.008953 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.010995 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.011002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.011022 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.011368 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.011697 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.011699 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.011700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.011856 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.012370 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.012380 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.012663 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.012843 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.013308 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":125,"threads_started":1}
I20260926 20:10:53.015933 30227 heartbeater.cc:347] Connected to a master server at 127.23.50.190:41147
I20260926 20:10:53.016211 30227 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:53.016778 30227 heartbeater.cc:511] Master 127.23.50.190:41147 requested a full tablet report, sending...
I20260926 20:10:53.017930 29902 ts_manager.cc:194] Registered new tserver with Master: c7758e9f4ea346a394c342bea4de2728 (127.23.50.130:37165)
I20260926 20:10:53.018822 29902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:50109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab.
Entry for principal oryx/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab.
I20260926 20:10:53.035213 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:41147
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab
--principal=oryx/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:53.153537 30232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:53.153882 30232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:53.153932 30232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:53.158243 30232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:53.158317 30232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:53.158380 30232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:10:53.163985 30232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab
--principal=oryx/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:41147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:53.165072 30232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:53.166152 30232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:53.176103 30238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:53.176113 30237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:53.176177 30232 server_base.cc:1034] running on GCE node
W20260926 20:10:53.176113 30240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:53.176934 30232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:53.177662 30232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:53.178835 30232 hybrid_clock.cc:648] HybridClock initialized: now 1790453453178798 us; error 51 us; skew 500 ppm
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453453, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:53.181816 30232 init.cc:377] Logged in from keytab as oryx/127.23.50.131@KRBTEST.COM (short username oryx)
I20260926 20:10:53.182824 30232 webserver.cc:460] Webserver started at http://127.23.50.131:43139/ using document root <none> and password file <none>
I20260926 20:10:53.183403 30232 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:53.183493 30232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:53.183718 30232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:53.185814 30232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "28732512ad3746219c398c03e2602ae5"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "e202be93524c23c1da47b2293e825ad9"
server_key_iv: "c32f15219bbb62a9cc79232e32e930ea"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.186434 30232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "28732512ad3746219c398c03e2602ae5"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "e202be93524c23c1da47b2293e825ad9"
server_key_iv: "c32f15219bbb62a9cc79232e32e930ea"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.190339 30232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:10:53.193044 30247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:53.194378 30232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:53.194517 30232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "28732512ad3746219c398c03e2602ae5"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "e202be93524c23c1da47b2293e825ad9"
server_key_iv: "c32f15219bbb62a9cc79232e32e930ea"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.194638 30232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:53.207261 30232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:53.210825 30232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:53.211045 30232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:53.211714 30232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:53.212733 30232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:53.212809 30232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:53.212882 30232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:53.212914 30232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:53.222232 30232 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:34899
I20260926 20:10:53.222283 30360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:34899 every 8 connection(s)
I20260926 20:10:53.223210 30232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:10:53.232291 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 30232
I20260926 20:10:53.232432 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:10:53.232795 23754 external_mini_cluster.cc:1428] Setting key c82894b9786609ebf06d980314a870f3
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453453, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
I20260926 20:10:53.238234 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.225177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:34999 (local address 127.23.50.190:41147)
0926 20:10:53.225343 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.225347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.226493 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 20:10:53.227932 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.227935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.227997 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.228121 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.229377 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.229999 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.231215 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.231459 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.231604 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.234642 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.234645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.234647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.234674 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.236067 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.236616 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.236624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.236626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.236710 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.237087 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.237091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.237093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.237292 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.237398 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.237943 (+ 545us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.238071 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:10:53.239148 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.225648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.131:34999)
0926 20:10:53.226232 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.226278 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.227697 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.228317 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.228325 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.228705 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.229211 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.229225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.230131 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.230145 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.231048 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.231057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.231651 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.231655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.231845 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.232546 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.232570 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.234469 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.236197 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.236212 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.236230 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.236499 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.236803 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.236807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.236809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.236985 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.237396 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.237403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.237676 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.237897 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.238276 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":188,"threads_started":1}
I20260926 20:10:53.240351 30361 heartbeater.cc:347] Connected to a master server at 127.23.50.190:41147
I20260926 20:10:53.240638 30361 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:53.241251 30361 heartbeater.cc:511] Master 127.23.50.190:41147 requested a full tablet report, sending...
I20260926 20:10:53.242538 29902 ts_manager.cc:194] Registered new tserver with Master: 28732512ad3746219c398c03e2602ae5 (127.23.50.131:34899)
I20260926 20:10:53.243085 29902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:34999
I20260926 20:10:53.247329 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.50.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.189.keytab.
Entry for principal oryx/127.23.50.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.189.keytab.
I20260926 20:10:53.278092 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.189:40407
--webserver_interface=127.23.50.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.189.keytab
--principal=oryx/127.23.50.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:41147,127.23.50.189:40407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:53.396719 30366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:53.397017 30366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:53.397058 30366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:53.401618 30366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:53.401706 30366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:53.401729 30366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:53.401747 30366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:53.401763 30366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:53.407629 30366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:41147,127.23.50.189:40407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.189.keytab
--principal=oryx/127.23.50.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.189:40407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.50.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
W20260926 20:10:53.407918 30366 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.50.190:41147,127.23.50.189:40407'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 20:10:53.408941 30366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:53.410174 30366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:53.420349 30374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:53.420394 30366 server_base.cc:1034] running on GCE node
W20260926 20:10:53.420353 30371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:53.420349 30372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:53.421244 30366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:53.422505 30366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:53.423732 30366 hybrid_clock.cc:648] HybridClock initialized: now 1790453453423711 us; error 33 us; skew 500 ppm
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453453, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:53.426851 30366 init.cc:377] Logged in from keytab as oryx/127.23.50.189@KRBTEST.COM (short username oryx)
I20260926 20:10:53.427804 30366 webserver.cc:460] Webserver started at http://127.23.50.189:43761/ using document root <none> and password file <none>
I20260926 20:10:53.428385 30366 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:53.428452 30366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:53.428630 30366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:53.430951 30366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data/instance:
uuid: "873b624c07244e059d19b23dbc5b9b88"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "ecfa9c3fb72a07d46b065095440c4a53"
server_key_iv: "9194e73d374f21ebdf5f5f7a604085c9"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.431555 30366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/wal/instance:
uuid: "873b624c07244e059d19b23dbc5b9b88"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "ecfa9c3fb72a07d46b065095440c4a53"
server_key_iv: "9194e73d374f21ebdf5f5f7a604085c9"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.435824 30366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 20:10:53.438479 30381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:53.440088 30366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:53.440209 30366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/wal
uuid: "873b624c07244e059d19b23dbc5b9b88"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "ecfa9c3fb72a07d46b065095440c4a53"
server_key_iv: "9194e73d374f21ebdf5f5f7a604085c9"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.440354 30366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:53.451890 30366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:53.455313 30366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:53.455523 30366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:53.462842 30366 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.189:40407
I20260926 20:10:53.463179 30433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.189:40407 every 8 connection(s)
I20260926 20:10:53.463919 30366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
I20260926 20:10:53.464942 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 30366
I20260926 20:10:53.468315 30434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:53.472935 30434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.50.189@KRBTEST.COM
I20260926 20:10:53.479943 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.466509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.23.50.189:40407)
0926 20:10:53.466938 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.466953 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.466994 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:10:53.467653 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.467670 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.468156 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.468526 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.469426 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.470591 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.471527 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.472034 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.472209 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.475132 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.475138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.475157 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.475193 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.476953 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.477418 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.477429 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.477433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.477544 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.477869 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.477875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.477877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.478288 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.478525 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.478769 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.479053 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":290,"thread_start_us":140,"threads_started":1}
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453453, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.189@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
I20260926 20:10:53.484412 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.474995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.23.50.190:41147)
0926 20:10:53.475151 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.475156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.475718 (+ 562us) server_negotiation.cc:407] Connection header received
0926 20:10:53.475767 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.475769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.475816 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.475920 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.476666 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.477240 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.478322 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.478571 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.478721 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.481061 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.481065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.481066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.481101 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.482596 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.483068 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.483071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.483073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.483157 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.483434 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.483438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.483440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.483671 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.483785 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.484116 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.484241 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:10:53.484422 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.474941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:38536)
0926 20:10:53.475374 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.475429 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.475558 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.475964 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.475970 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.476386 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.476514 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.476520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.477382 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.477385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.478189 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.478199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.478736 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.478740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.478866 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.479511 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.479523 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.480900 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.482720 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.482731 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.482736 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.482958 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.483251 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.483255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.483257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.483342 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.483779 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.483785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.483904 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.484059 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.484242 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":116,"threads_started":1}
I20260926 20:10:53.485724 30434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453453, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.189@KRBTEST.COM for oryx/127.23.50.189@KRBTEST.COM
I20260926 20:10:53.494194 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.486358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.23.50.189:40407)
0926 20:10:53.486465 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.486469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.486483 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:10:53.486519 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.486525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.486566 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.486673 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.487039 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.487973 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.488712 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.488941 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.489017 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.491038 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.491041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.491043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.491063 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.492481 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.492952 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.492956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.492958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.493028 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.493296 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.493299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.493301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.493544 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.493634 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.493918 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.494065 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":51}
I20260926 20:10:53.494195 30438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.486228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:40407 (local address 127.0.0.1:45802)
0926 20:10:53.486345 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.486360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.486454 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.486668 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.486670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.486808 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.486906 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.486911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.488089 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.488102 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.488608 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.488616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.489035 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.489037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.489109 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.489542 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.489554 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.490889 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.492622 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.492653 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.492655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.492839 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.493117 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.493122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.493124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.493200 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.493655 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.493657 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.493735 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.493838 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.494012 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 20:10:53.497483 30434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: Bootstrap starting.
I20260926 20:10:53.500345 30434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:53.501183 30434 log.cc:824] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:53.503367 30434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: No bootstrap required, opened a new log
I20260926 20:10:53.506493 30434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } }
I20260926 20:10:53.506721 30434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:53.506805 30434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 873b624c07244e059d19b23dbc5b9b88, State: Initialized, Role: FOLLOWER
I20260926 20:10:53.507241 30434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } }
I20260926 20:10:53.507787 30440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } } }
I20260926 20:10:53.507911 30440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:10:53.508486 30434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:10:53.513470 30451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:10:53.513566 30451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:10:53.515614 30366 master_runner.cc:428] Detected that this master 873b624c07244e059d19b23dbc5b9b88 is joining an existing cluster
I20260926 20:10:53.515709 30366 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.50.189:40407
I20260926 20:10:53.527268 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.518562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.23.50.190:41147)
0926 20:10:53.518716 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.518720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.518955 (+ 235us) server_negotiation.cc:407] Connection header received
0926 20:10:53.519084 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.519088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.519145 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.519240 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.520032 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.520801 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.521673 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.521976 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.522088 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.523537 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.523540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.523543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.523575 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.525422 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.526033 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.526039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.526042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.526135 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.526434 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.526439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.526440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.526631 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.526743 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.527005 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.527123 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 20:10:53.527293 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.518390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:38538)
0926 20:10:53.518846 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.518862 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.518977 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.519385 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.519388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.519720 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.519854 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.519862 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.520921 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.520924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.521553 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.521562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.522224 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.522227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.522315 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.522861 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.522883 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.523394 (+ 511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.525544 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.525547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.525550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.525928 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.526239 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.526244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.526246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.526342 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.526736 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.526739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.526837 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.526940 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.527083 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":23,"spinlock_wait_cycles":1792,"thread_start_us":102,"threads_started":1}
I20260926 20:10:53.527503 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.518479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45814 (local address 127.23.50.189:40407)
0926 20:10:53.518607 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.518614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.518986 (+ 372us) server_negotiation.cc:407] Connection header received
0926 20:10:53.519159 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.519162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.519245 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.519321 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.519916 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.520700 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.522063 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.522366 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.522484 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.523774 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.523780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.523786 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.523816 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.525561 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.526100 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.526104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.526106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.526194 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.526501 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.526505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.526506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.526723 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.526805 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.527137 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.527283 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49,"spinlock_wait_cycles":768}
I20260926 20:10:53.527999 30457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.518390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:40407 (local address 127.0.0.1:45814)
0926 20:10:53.518917 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.518930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.519033 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.519422 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.519425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.519696 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.519803 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.519808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.520838 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.520845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.521942 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.521954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.522498 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.522503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.522617 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.523020 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.523040 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.523580 (+ 540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.525670 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.525676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.525679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.525972 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.526306 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.526309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.526311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.526400 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.526871 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.526874 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.526966 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.527653 (+ 687us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.527801 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":26,"thread_start_us":82,"threads_started":1}
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453452, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.190@KRBTEST.COM for oryx/127.23.50.189@KRBTEST.COM
I20260926 20:10:53.544092 30437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.532643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.23.50.189:40407)
0926 20:10:53.532783 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.532788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.534302 (+ 1514us) server_negotiation.cc:407] Connection header received
0926 20:10:53.534528 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.534531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.534590 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.534680 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.535497 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.536346 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.537329 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.537562 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.537674 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.540213 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.540217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.540219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.540243 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.541828 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.542453 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.542459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.542461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.542552 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.542899 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.542904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.542906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.543110 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.543288 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.543770 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.543910 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:10:53.544111 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.532608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:40407 (local address 127.0.0.1:45820)
0926 20:10:53.534166 (+ 1558us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.534215 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.534387 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.534724 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.534731 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.535122 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.535339 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.535350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.536503 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.536509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.537204 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.537213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.537680 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.537684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.537809 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.538477 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.538490 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.540039 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.542030 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.542041 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.542050 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.542332 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.542689 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.542692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.542695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.542793 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.543285 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.543294 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.543444 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.543691 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.543934 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1389,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for oryx/127.23.50.188@KRBTEST.COM; defaulting to no policy
I20260926 20:10:53.546321 29902 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.50.189:40407: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "873b624c07244e059d19b23dbc5b9b88"
member_type: NON_VOTER
last_known_addr {
host: "127.23.50.189"
port: 40407
}
attrs {
promote: true
}
}
dest_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e"
cas_config_opid_index: -1
I20260926 20:10:53.547266 29902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.188@KRBTEST.COM" created.
I20260926 20:10:53.549723 30464 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 873b624c07244e059d19b23dbc5b9b88 (127.23.50.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true } } }
I20260926 20:10:53.550640 30408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:53.551759 30464 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.50.189:40407
I20260926 20:10:53.552156 30408 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 FOLLOWER]: Refusing update from remote peer 8c0a1ed2d1614a04add0c82a6c22d24e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 20:10:53.552322 30366 master.cc:537] Master@127.23.50.189:40407 shutting down...
I20260926 20:10:53.553201 30465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [LEADER]: Connected to new peer: Peer: permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:10:53.555372 29889 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 20:10:53.556509 29889 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e -> Peer 873b624c07244e059d19b23dbc5b9b88 (127.23.50.189:40407): Couldn't send request to peer 873b624c07244e059d19b23dbc5b9b88. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:10:53.557220 30462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true } } }
I20260926 20:10:53.557353 30462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:53.558166 30366 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:53.558472 30366 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:53.558596 30366 tablet_replica.cc:330] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.188.keytab.
Entry for principal oryx/127.23.50.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.188.keytab.
I20260926 20:10:53.569751 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.188:45035
--webserver_interface=127.23.50.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.188.keytab
--principal=oryx/127.23.50.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:41147,127.23.50.189:40407,127.23.50.188:45035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:10:53.573751 30366 master.cc:552] Master@127.23.50.189:40407 shutdown complete.
I20260926 20:10:53.573828 30366 master_runner.cc:305] Clearing existing system tablet
I20260926 20:10:53.574198 30366 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:53.574265 30366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:53.575770 30366 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:10:53.576846 30470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:53.577148 30366 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:53.577288 30366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:10:53.577325 30366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/wal
uuid: "873b624c07244e059d19b23dbc5b9b88"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "ecfa9c3fb72a07d46b065095440c4a53"
server_key_iv: "9194e73d374f21ebdf5f5f7a604085c9"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.579785 30366 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:53.582542 30366 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:10:53.582657 30366 log.cc:1196] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 20:10:53.582958 30366 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: Deleting consensus metadata
I20260926 20:10:53.583160 30366 master_runner.cc:315] Copying system tablet from 127.23.50.190:41147
I20260926 20:10:53.584486 30366 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: tablet copy: Beginning tablet copy session from remote peer at address 127.23.50.190:41147
I20260926 20:10:53.594769 30476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.585151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:38548)
0926 20:10:53.585468 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.585483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.585606 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.585980 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.585985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.586225 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.586406 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.586418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.587635 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.587638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.588632 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.588644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.589342 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.589346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.589449 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.590127 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.590145 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.590596 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.592798 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.592801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.592804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.593067 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.593394 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.593398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.593400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.593498 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.594172 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.594175 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.594284 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.594399 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.594559 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":88,"threads_started":1}
I20260926 20:10:53.594812 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.585243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.23.50.190:41147)
0926 20:10:53.585384 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.585389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.585576 (+ 187us) server_negotiation.cc:407] Connection header received
0926 20:10:53.585704 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.585708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.585765 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.585958 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.586555 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.587496 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.588795 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.589207 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.589330 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.590770 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.590774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.590777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.590810 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.592646 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.593182 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.593186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.593188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.593288 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.593748 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.593752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.593754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.594059 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.594174 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.594485 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.594625 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:10:53.595888 29922 tablet_copy_service.cc:140] P 8c0a1ed2d1614a04add0c82a6c22d24e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 873b624c07244e059d19b23dbc5b9b88 ({username='oryx', principal='oryx/127.23.50.189@KRBTEST.COM'} at 127.0.0.1:38548)
I20260926 20:10:53.596074 29922 tablet_copy_service.cc:161] P 8c0a1ed2d1614a04add0c82a6c22d24e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 873b624c07244e059d19b23dbc5b9b88 at {username='oryx', principal='oryx/127.23.50.189@KRBTEST.COM'} at 127.0.0.1:38548: session id = 873b624c07244e059d19b23dbc5b9b88-00000000000000000000000000000000
I20260926 20:10:53.598042 29922 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:10:53.599931 30366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:10:53.602555 30366 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: tablet copy: Starting download of 0 data blocks...
I20260926 20:10:53.602876 30366 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: tablet copy: Starting download of 1 WAL segments...
I20260926 20:10:53.605042 30366 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:10:53.607204 29922 tablet_copy_service.cc:342] P 8c0a1ed2d1614a04add0c82a6c22d24e: Request end of tablet copy session 873b624c07244e059d19b23dbc5b9b88-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.50.189@KRBTEST.COM'} at 127.0.0.1:38548
I20260926 20:10:53.607352 29922 tablet_copy_service.cc:433] P 8c0a1ed2d1614a04add0c82a6c22d24e: ending tablet copy session 873b624c07244e059d19b23dbc5b9b88-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 873b624c07244e059d19b23dbc5b9b88
I20260926 20:10:53.617540 30366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:53.619508 30477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:53.622967 30478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:53.623281 30366 server_base.cc:1034] running on GCE node
W20260926 20:10:53.626044 30480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:53.626363 30366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:53.626852 30366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:53.628029 30366 hybrid_clock.cc:648] HybridClock initialized: now 1790453453628004 us; error 48 us; skew 500 ppm
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453453, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:53.629700 30366 init.cc:377] Logged in from keytab as oryx/127.23.50.189@KRBTEST.COM (short username oryx)
I20260926 20:10:53.630293 30366 webserver.cc:460] Webserver started at http://127.23.50.189:36131/ using document root <none> and password file <none>
I20260926 20:10:53.630492 30366 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:53.630535 30366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:53.632055 30366 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:10:53.633119 30487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:53.633378 30366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 20:10:53.633455 30366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/wal
uuid: "873b624c07244e059d19b23dbc5b9b88"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "ecfa9c3fb72a07d46b065095440c4a53"
server_key_iv: "9194e73d374f21ebdf5f5f7a604085c9"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.633531 30366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:53.641084 30366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:53.643270 30366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:53.648242 30366 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.189:40407
I20260926 20:10:53.648867 30366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
I20260926 20:10:53.649289 30539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.189:40407 every 8 connection(s)
I20260926 20:10:53.650831 30540 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:10:53.651896 30540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: Bootstrap starting.
I20260926 20:10:53.662797 30540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:10:53.663163 30540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: Bootstrap complete.
I20260926 20:10:53.663811 30540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true } }
I20260926 20:10:53.663957 30540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 873b624c07244e059d19b23dbc5b9b88, State: Initialized, Role: LEARNER
I20260926 20:10:53.664062 30540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true } }
I20260926 20:10:53.664479 30543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true } } }
I20260926 20:10:53.664644 30543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:10:53.664546 30540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:53.667811 30554 catalog_manager.cc:1259] Loaded cluster ID: d249234698a84df49b235ad8f79ffee2
I20260926 20:10:53.667938 30554 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: loading cluster ID for follower catalog manager: success
I20260926 20:10:53.670374 30554 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: acquiring CA information for follower catalog manager: success
I20260926 20:10:53.671852 30554 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 20:10:53.710994 30468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:53.711369 30468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:53.711457 30468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:53.716135 30468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:53.716253 30468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:53.716289 30468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:53.716328 30468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:53.716385 30468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:53.722659 30468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:41147,127.23.50.189:40407,127.23.50.188:45035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.188.keytab
--principal=oryx/127.23.50.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.188:45035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.50.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:53.723948 30468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:53.725121 30468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:53.734169 30560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:53.734153 30562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:53.734153 30559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:53.734838 30468 server_base.cc:1034] running on GCE node
I20260926 20:10:53.735448 30468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:53.736656 30468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:53.737939 30468 hybrid_clock.cc:648] HybridClock initialized: now 1790453453737920 us; error 33 us; skew 500 ppm
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453453, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:53.741295 30468 init.cc:377] Logged in from keytab as oryx/127.23.50.188@KRBTEST.COM (short username oryx)
I20260926 20:10:53.742537 30468 webserver.cc:460] Webserver started at http://127.23.50.188:38539/ using document root <none> and password file <none>
I20260926 20:10:53.743126 30468 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:53.743201 30468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:53.743426 30468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:53.745684 30468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data/instance:
uuid: "e0dca02309bf476d8f5a04655fc1bf13"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "2338797f01a07e6eb5c005ecb8fd03dd"
server_key_iv: "eff1509c47a40848c7aa71daa8688060"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.746439 30468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/wal/instance:
uuid: "e0dca02309bf476d8f5a04655fc1bf13"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "2338797f01a07e6eb5c005ecb8fd03dd"
server_key_iv: "eff1509c47a40848c7aa71daa8688060"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.750881 30468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:10:53.753630 30569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:53.755062 30468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:53.755178 30468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/wal
uuid: "e0dca02309bf476d8f5a04655fc1bf13"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "2338797f01a07e6eb5c005ecb8fd03dd"
server_key_iv: "eff1509c47a40848c7aa71daa8688060"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.755331 30468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:53.768857 30093 heartbeater.cc:503] Master 127.23.50.190:41147 was elected leader, sending a full tablet report...
I20260926 20:10:53.771929 30468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:53.775417 30468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:53.775669 30468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:53.783358 30468 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.188:45035
I20260926 20:10:53.783663 30621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.188:45035 every 8 connection(s)
I20260926 20:10:53.784487 30468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
I20260926 20:10:53.787429 30622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:53.790638 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 30468
I20260926 20:10:53.791383 30622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453453, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.188@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.50.188@KRBTEST.COM
I20260926 20:10:53.804507 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.793388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.23.50.190:41147)
0926 20:10:53.793534 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.793539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.793837 (+ 298us) server_negotiation.cc:407] Connection header received
0926 20:10:53.794139 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.794145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.794232 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.794296 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.795195 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.795739 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.796813 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.797039 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.797156 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.800424 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.800429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.800431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.800467 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.802207 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.802907 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.802910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.802911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.802985 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.803366 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.803370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.803371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.803568 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.803678 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.804199 (+ 521us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.804336 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:10:53.805457 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.793141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:38562)
0926 20:10:53.793679 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.793729 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.793910 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.794462 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.794466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.794802 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.795048 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.795061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.795876 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.795880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.796678 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.796687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.797192 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.797196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.797357 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.798027 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.798053 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.800196 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.802332 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.802342 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.802356 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.802797 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.803085 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.803090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.803092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.803251 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.803675 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.803683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.803961 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.804160 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.804493 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":133,"threads_started":1}
I20260926 20:10:53.805461 30624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.792174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.23.50.188:45035)
0926 20:10:53.792610 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.792625 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.792659 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:10:53.793479 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.793504 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.793932 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.794277 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.795079 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.795952 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.796832 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.797204 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.797364 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.800868 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.800874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.800887 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.800911 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.802968 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.803412 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.803419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.803420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.803500 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.803793 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.803798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.803799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.804046 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.804271 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.804467 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.804609 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"spinlock_wait_cycles":9984,"thread_start_us":148,"threads_started":1}
I20260926 20:10:53.806908 30622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453453, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.188@KRBTEST.COM for oryx/127.23.50.189@KRBTEST.COM
I20260926 20:10:53.815632 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.807435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:40407 (local address 127.0.0.1:45830)
0926 20:10:53.807555 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.807569 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.807644 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.808075 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.808078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.808281 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.808377 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.808383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.809302 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.809306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.809947 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.809956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.810419 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.810423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.810526 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.811028 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.811042 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.812353 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.814341 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.814343 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.814345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.814533 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.814793 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.814795 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.814796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.814866 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.815213 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.815215 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.815309 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.815395 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.815507 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 20:10:53.815819 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.807546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.23.50.189:40407)
0926 20:10:53.807840 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.807844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.807855 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:10:53.807891 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.807893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.807955 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.808063 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.808495 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.809187 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.810061 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.810303 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.810420 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.812504 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.812511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.812513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.812594 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.814240 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.814617 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.814619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.814620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.814693 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.814941 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.814943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.814944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.815132 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.815208 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.815507 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.815628 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
I20260926 20:10:53.816563 30622 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453453, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.188@KRBTEST.COM for oryx/127.23.50.188@KRBTEST.COM
I20260926 20:10:53.824947 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.817091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:45035 (local address 127.0.0.1:52854)
0926 20:10:53.817270 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.817286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.817386 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.817614 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.817617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.817790 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.817921 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.817929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.818668 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.818671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.819173 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.819180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.819635 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.819638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.819732 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.820180 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.820190 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.821667 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.823381 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.823386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.823388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.823673 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.823948 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.823951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.823953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.824031 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.824407 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.824410 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.824487 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.824601 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.824753 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":4}
I20260926 20:10:53.824946 30624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.817126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.23.50.188:45035)
0926 20:10:53.817292 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.817298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.817335 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:10:53.817476 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.817480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.817515 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.817604 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.818035 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.818550 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.819295 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.819532 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.819638 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.821802 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.821805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.821808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.821835 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.823248 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.823767 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.823773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.823775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.823852 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.824126 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.824129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.824131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.824312 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.824418 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.824670 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.824808 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":64}
I20260926 20:10:53.828593 30622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: Bootstrap starting.
I20260926 20:10:53.831015 30622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:53.831732 30622 log.cc:824] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:53.833834 30622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: No bootstrap required, opened a new log
I20260926 20:10:53.836963 30622 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } }
I20260926 20:10:53.837172 30622 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:53.837203 30622 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0dca02309bf476d8f5a04655fc1bf13, State: Initialized, Role: FOLLOWER
I20260926 20:10:53.837648 30622 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } }
I20260926 20:10:53.838193 30628 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } } }
I20260926 20:10:53.838339 30628 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:10:53.838963 30622 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:10:53.843786 30639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:10:53.843878 30639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:10:53.847414 30468 master_runner.cc:428] Detected that this master e0dca02309bf476d8f5a04655fc1bf13 is joining an existing cluster
I20260926 20:10:53.847489 30468 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.50.188:45035
I20260926 20:10:53.859220 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 29964
I20260926 20:10:53.859414 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.850519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.23.50.190:41147)
0926 20:10:53.850640 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.850644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.850657 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:10:53.850715 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.850718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.850767 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.850854 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.851596 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.852398 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.853919 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.854268 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.854337 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.855855 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.855859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.855862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.855891 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.857495 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.857968 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.857971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.857973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.858062 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.858457 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.858460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.858462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.858653 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.858773 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.859074 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.859200 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:10:53.860651 30645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.849696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:38568)
0926 20:10:53.850189 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.850204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.850302 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.850978 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.850980 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.851292 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.851449 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.851456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.852522 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.852525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.853753 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.853763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.854493 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.854497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.854691 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.855287 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.855298 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.855704 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.857605 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.857608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.857610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.857833 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.858161 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.858164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.858166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.858269 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.858758 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.858761 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.858861 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.860369 (+ 1508us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.860491 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":208,"thread_start_us":99,"threads_started":1}
I20260926 20:10:53.865851 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:36633
--local_ip_for_outbound_sockets=127.23.50.129
--tserver_master_addrs=127.23.50.190:41147
--webserver_port=41235
--webserver_interface=127.23.50.129
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab
--principal=oryx/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.50.190:41147,127.23.50.189:40407
--tserver_master_addrs=127.23.50.190:41147,127.23.50.189:40407,127.23.50.188:45035 with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453452, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.190@KRBTEST.COM for oryx/127.23.50.188@KRBTEST.COM
I20260926 20:10:53.881886 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.864331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:45035 (local address 127.0.0.1:52884)
0926 20:10:53.864493 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.864511 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.864599 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.865344 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.865347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.865543 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.865651 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.865658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.866864 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.866868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.867737 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.867747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.868402 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.868406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.868512 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.869027 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.869047 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.870864 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.880020 (+ 9156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.880025 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.880027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.880316 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.880738 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.880741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.880743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.880844 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.881300 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.881303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.881410 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.881489 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.881656 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 20:10:53.882876 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.864663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.23.50.188:45035)
0926 20:10:53.865024 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.865035 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.865051 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:10:53.865127 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.865133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.865210 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.865306 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.865772 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.866745 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.867857 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.868270 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.868395 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.871059 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.871066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.871071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.871104 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.879824 (+ 8720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.880545 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.880548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.880550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.880635 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.880956 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.880960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.880962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.881192 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.882480 (+ 1288us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.882558 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.882683 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260926 20:10:53.884114 29902 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.50.188:45035: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13"
member_type: NON_VOTER
last_known_addr {
host: "127.23.50.188"
port: 45035
}
attrs {
promote: true
}
}
dest_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e"
cas_config_opid_index: 5
I20260926 20:10:53.884784 29902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } }
I20260926 20:10:53.886250 30462 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e0dca02309bf476d8f5a04655fc1bf13 (127.23.50.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:53.887878 30462 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.50.188:45035
I20260926 20:10:53.888082 30462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:53.888214 30462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:53.888711 30468 master.cc:537] Master@127.23.50.188:45035 shutting down...
I20260926 20:10:53.889427 30595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:53.891036 30595 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 FOLLOWER]: Refusing update from remote peer 8c0a1ed2d1614a04add0c82a6c22d24e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:10:53.892069 30465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:10:53.895102 29886 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e -> Peer e0dca02309bf476d8f5a04655fc1bf13 (127.23.50.188:45035): Couldn't send request to peer e0dca02309bf476d8f5a04655fc1bf13. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 20:10:53.895812 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.850187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:45035 (local address 127.0.0.1:52870)
0926 20:10:53.850797 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.850812 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.850907 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.851226 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.851228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.851410 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.851529 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.851536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.852522 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.852525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.853828 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.853837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.854493 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.854497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.854749 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.880889 (+ 26140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.880904 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.881323 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.884911 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.884915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.884917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.885196 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.888906 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.888910 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.888913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.889027 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.895247 (+ 6220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.895250 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.895367 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.895467 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.895628 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":171,"thread_start_us":87,"threads_started":1}
I20260926 20:10:53.895824 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.886430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:40407 (local address 127.0.0.1:45852)
0926 20:10:53.886979 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.886994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.887083 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.887640 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.887644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.887662 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:53.887771 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.887777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.892589 (+ 4812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.892593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.893933 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.893946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.895388 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.895393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.895500 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.895564 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.895613 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.895696 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463}
I20260926 20:10:53.897217 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.849751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:40407 (local address 127.0.0.1:45838)
0926 20:10:53.850383 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.850396 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.850529 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.850909 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.850912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.851261 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.851418 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.851426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.852539 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.852543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.853704 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.853715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.854507 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.854510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.854668 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.879683 (+ 25015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.879703 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.880169 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.885301 (+ 5132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.885303 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.885305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.885513 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.891027 (+ 5514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.891032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.891036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.891143 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.896664 (+ 5521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.896667 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.896775 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.896874 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.897022 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":129,"spinlock_wait_cycles":7424,"thread_start_us":123,"threads_started":1}
I20260926 20:10:53.897603 30650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.886663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.23.50.189:40407)
0926 20:10:53.887347 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.887351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.887364 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:10:53.887413 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.887415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.887477 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.887882 (+ 405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:53.888047 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.892447 (+ 4400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.894470 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.895152 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.896967 (+ 1815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.897121 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.897171 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.897471 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":71,"threads_started":1}
I20260926 20:10:53.897702 30468 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:53.898037 30468 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:53.898144 30468 tablet_replica.cc:330] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: stopping tablet replica
I20260926 20:10:53.898247 30506 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 LEARNER]: Refusing update from remote peer 8c0a1ed2d1614a04add0c82a6c22d24e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 20:10:53.898972 30624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.850553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.23.50.188:45035)
0926 20:10:53.850637 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.850642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.850863 (+ 221us) server_negotiation.cc:407] Connection header received
0926 20:10:53.850988 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.850990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.851027 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.851074 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.851640 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.852397 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.853959 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.854268 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.854337 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.881467 (+ 27130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.881470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.881473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.881492 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.883751 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.885603 (+ 1852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.885606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.885608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.885757 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.891427 (+ 5670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.891434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.891437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.891689 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.891788 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.898694 (+ 6906us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.898816 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 20:10:53.899323 30465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [LEADER]: Connected to new peer: Peer: permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 20:10:53.900559 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.849864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45838 (local address 127.23.50.189:40407)
0926 20:10:53.849994 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.850001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.850466 (+ 465us) server_negotiation.cc:407] Connection header received
0926 20:10:53.850653 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.850656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.850712 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.850764 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.851557 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.852395 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.853880 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.854255 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.854335 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.880386 (+ 26051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.880390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.880393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.880426 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.884773 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.890783 (+ 6010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.890789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.890793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.890902 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.896106 (+ 5204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.896110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.896112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.896362 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.896453 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.900278 (+ 3825us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.900425 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:10:53.901288 30506 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e0dca02309bf476d8f5a04655fc1bf13 (127.23.50.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:53.903921 30543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c0a1ed2d1614a04add0c82a6c22d24e. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:53.904086 30543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:10:53.904181 30543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: NON_VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: true } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:53.904242 30543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:10:53.904582 30468 master.cc:552] Master@127.23.50.188:45035 shutdown complete.
I20260926 20:10:53.904630 30468 master_runner.cc:305] Clearing existing system tablet
I20260926 20:10:53.904879 30468 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:53.904943 30468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:53.905475 30465 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e: attempting to promote NON_VOTER 873b624c07244e059d19b23dbc5b9b88 to VOTER
I20260926 20:10:53.905972 30465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } }
I20260926 20:10:53.906390 30468 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 20:10:53.907451 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.907157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:45035 (local address 127.0.0.1:52900)
0926 20:10:53.907306 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.907377 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:45035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 20:10:53.907596 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.907192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:45035 (local address 127.0.0.1:52916)
0926 20:10:53.907447 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.907512 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:45035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 20:10:53.907902 30506 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 LEARNER]: Refusing update from remote peer 8c0a1ed2d1614a04add0c82a6c22d24e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 20:10:53.908381 30654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:53.908564 30468 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:53.908541 30464 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [LEADER]: Connected to new peer: Peer: permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 20:10:53.908684 30468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:10:53.908720 30468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/wal
uuid: "e0dca02309bf476d8f5a04655fc1bf13"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "2338797f01a07e6eb5c005ecb8fd03dd"
server_key_iv: "eff1509c47a40848c7aa71daa8688060"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.910086 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.909817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:45035 (local address 127.0.0.1:52928)
0926 20:10:53.909971 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.910019 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:45035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260926 20:10:53.910211 29886 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e -> Peer e0dca02309bf476d8f5a04655fc1bf13 (127.23.50.188:45035): Couldn't send request to peer e0dca02309bf476d8f5a04655fc1bf13. Status: Network error: Client connection negotiation failed: client connection to 127.23.50.188:45035: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 20:10:53.910764 30468 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:53.910786 30464 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 873b624c07244e059d19b23dbc5b9b88 (127.23.50.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:53.912211 30464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:53.912321 30464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:53.912434 30464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:53.912505 30464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:53.913146 30468 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 20:10:53.913311 30468 log.cc:1196] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 20:10:53.913173 30506 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 873b624c07244e059d19b23dbc5b9b88 (127.23.50.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:53.913568 30468 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: Deleting consensus metadata
I20260926 20:10:53.913973 30468 master_runner.cc:315] Copying system tablet from 127.23.50.190:41147
I20260926 20:10:53.915089 30468 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: tablet copy: Beginning tablet copy session from remote peer at address 127.23.50.190:41147
I20260926 20:10:53.915248 30652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:53.915377 30652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:10:53.924832 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.915832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.23.50.190:41147)
0926 20:10:53.915938 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:53.915942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:53.916164 (+ 222us) server_negotiation.cc:407] Connection header received
0926 20:10:53.916314 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:53.916317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:53.916367 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:53.916452 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:53.917042 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.917888 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.919024 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.919389 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.919494 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.920984 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:53.920988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:53.920991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:53.921020 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:53.922763 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.923247 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.923259 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.923261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.923373 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:53.923685 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:53.923689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:53.923691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:53.923967 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:53.924077 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:53.924534 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 20:10:53.924669 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:10:53.925024 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:53.915750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:38584)
0926 20:10:53.916057 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:53.916071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:53.916214 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:53.916503 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:53.916506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:53.916745 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:53.916907 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.916918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.918017 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.918021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.918895 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:53.918907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:53.919521 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:53.919525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:53.919640 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:53.920357 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:53.920373 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:53.920826 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:53.922882 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.922886 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.922888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.923145 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.923482 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:53.923485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:53.923487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:53.923577 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:53.924167 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:53.924170 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:53.924279 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 20:10:53.924404 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 20:10:53.924569 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":95,"threads_started":1}
I20260926 20:10:53.925483 29922 tablet_copy_service.cc:140] P 8c0a1ed2d1614a04add0c82a6c22d24e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e0dca02309bf476d8f5a04655fc1bf13 ({username='oryx', principal='oryx/127.23.50.188@KRBTEST.COM'} at 127.0.0.1:38584)
I20260926 20:10:53.925592 29922 tablet_copy_service.cc:161] P 8c0a1ed2d1614a04add0c82a6c22d24e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e0dca02309bf476d8f5a04655fc1bf13 at {username='oryx', principal='oryx/127.23.50.188@KRBTEST.COM'} at 127.0.0.1:38584: session id = e0dca02309bf476d8f5a04655fc1bf13-00000000000000000000000000000000
I20260926 20:10:53.926738 29922 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 20:10:53.928243 30468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 20:10:53.931293 30468 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: tablet copy: Starting download of 0 data blocks...
I20260926 20:10:53.931538 30468 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: tablet copy: Starting download of 1 WAL segments...
I20260926 20:10:53.933518 30468 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 20:10:53.936139 29922 tablet_copy_service.cc:342] P 8c0a1ed2d1614a04add0c82a6c22d24e: Request end of tablet copy session e0dca02309bf476d8f5a04655fc1bf13-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.50.188@KRBTEST.COM'} at 127.0.0.1:38584
I20260926 20:10:53.936254 29922 tablet_copy_service.cc:433] P 8c0a1ed2d1614a04add0c82a6c22d24e: ending tablet copy session e0dca02309bf476d8f5a04655fc1bf13-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e0dca02309bf476d8f5a04655fc1bf13
I20260926 20:10:53.945674 30468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:53.947993 30661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:53.948048 30664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:53.948400 30468 server_base.cc:1034] running on GCE node
W20260926 20:10:53.948730 30662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:53.949049 30468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:53.949404 30468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:53.950557 30468 hybrid_clock.cc:648] HybridClock initialized: now 1790453453950542 us; error 31 us; skew 500 ppm
Sep 26 20:10:53 dist-test-slave-7034 krb5kdc[29857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453453, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:53.952641 30468 init.cc:377] Logged in from keytab as oryx/127.23.50.188@KRBTEST.COM (short username oryx)
I20260926 20:10:53.953225 30468 webserver.cc:460] Webserver started at http://127.23.50.188:39357/ using document root <none> and password file <none>
I20260926 20:10:53.953428 30468 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:53.953466 30468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:53.955049 30468 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:10:53.956192 30671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:53.956476 30468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 20:10:53.956598 30468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/wal
uuid: "e0dca02309bf476d8f5a04655fc1bf13"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "2338797f01a07e6eb5c005ecb8fd03dd"
server_key_iv: "eff1509c47a40848c7aa71daa8688060"
server_key_version: "encryptionkey@0"
I20260926 20:10:53.956765 30468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:53.978700 30468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:53.981473 30468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:53.987134 30468 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.188:45035
I20260926 20:10:53.987166 30723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.188:45035 every 8 connection(s)
I20260926 20:10:53.987979 30468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
I20260926 20:10:53.990123 30724 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:10:53.991266 30724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: Bootstrap starting.
I20260926 20:10:54.004184 30724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:10:54.004522 30724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: Bootstrap complete.
I20260926 20:10:54.005123 30724 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } }
I20260926 20:10:54.005265 30724 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e0dca02309bf476d8f5a04655fc1bf13, State: Initialized, Role: LEARNER
I20260926 20:10:54.005362 30724 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } }
I20260926 20:10:54.005831 30724 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:54.005745 30727 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:54.006088 30727 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:10:54.009496 30738 catalog_manager.cc:1259] Loaded cluster ID: d249234698a84df49b235ad8f79ffee2
I20260926 20:10:54.009615 30738 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: loading cluster ID for follower catalog manager: success
I20260926 20:10:54.012259 30738 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: acquiring CA information for follower catalog manager: success
I20260926 20:10:54.013916 30738 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:10:54.020996 30227 heartbeater.cc:503] Master 127.23.50.190:41147 was elected leader, sending a full tablet report...
W20260926 20:10:54.037755 30649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:54.038166 30649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:54.038270 30649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:54.042923 30649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:54.043036 30649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:54.043136 30649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:10:54.049808 30649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab
--principal=oryx/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:36633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=41235
--tserver_master_addrs=127.23.50.190:41147,127.23.50.189:40407,127.23.50.188:45035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:54.051198 30649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:54.052392 30649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:54.063763 30744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:54.063763 30743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:54.064095 30649 server_base.cc:1034] running on GCE node
W20260926 20:10:54.063817 30746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:54.064723 30649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:54.065588 30649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:54.066749 30649 hybrid_clock.cc:648] HybridClock initialized: now 1790453454066730 us; error 75 us; skew 500 ppm
Sep 26 20:10:54 dist-test-slave-7034 krb5kdc[29857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453454, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:54.070029 30649 init.cc:377] Logged in from keytab as oryx/127.23.50.129@KRBTEST.COM (short username oryx)
I20260926 20:10:54.070972 30649 webserver.cc:460] Webserver started at http://127.23.50.129:41235/ using document root <none> and password file <none>
I20260926 20:10:54.071570 30649 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:54.071686 30649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:54.075871 30649 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:10:54.078141 30753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:54.079499 30649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:10:54.079643 30649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "0c3b0b17121f483789d7ea584b9a5a79"
format_stamp: "Formatted at 2026-09-26 20:10:52 on dist-test-slave-7034"
server_key: "1255aab8bdecabdb518c2785befbfec7"
server_key_iv: "d0834b78491e180a8517977ced3483a5"
server_key_version: "encryptionkey@0"
I20260926 20:10:54.080217 30649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:54.098765 30649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:54.102555 30649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:54.102788 30649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:54.103457 30649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:54.104387 30649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:54.104460 30649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:54.104537 30649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:54.104580 30649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:54.114666 30649 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:36633
I20260926 20:10:54.114745 30866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:36633 every 8 connection(s)
I20260926 20:10:54.115772 30649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:10:54.118541 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 30649
I20260926 20:10:54.118758 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 30098
I20260926 20:10:54.129091 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:37165
--local_ip_for_outbound_sockets=127.23.50.130
--tserver_master_addrs=127.23.50.190:41147
--webserver_port=34051
--webserver_interface=127.23.50.130
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab
--principal=oryx/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.50.190:41147,127.23.50.189:40407
--tserver_master_addrs=127.23.50.190:41147,127.23.50.189:40407,127.23.50.188:45035 with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:10:54 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453454, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
Sep 26 20:10:54 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453454, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.189@KRBTEST.COM
Sep 26 20:10:54 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453454, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.188@KRBTEST.COM
I20260926 20:10:54.136855 30873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.118235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.129:57793)
0926 20:10:54.119604 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.119620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.120366 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.120924 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.120931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.121727 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:54.122678 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.122701 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.123943 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.123945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.125766 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.125795 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.126607 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.126611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.126958 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.128014 (+ 1056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:54.128029 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:54.130175 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:54.132678 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.132688 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.132712 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.133142 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.133541 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.133544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.133547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.133729 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.134368 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:54.134377 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:54.134859 (+ 482us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.135118 (+ 259us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.135568 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"mutex_wait_us":450,"spinlock_wait_cycles":92800,"thread_start_us":150,"threads_started":1}
I20260926 20:10:54.137647 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.118036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:34757 (local address 127.23.50.189:40407)
0926 20:10:54.118222 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.118228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.119020 (+ 792us) server_negotiation.cc:407] Connection header received
0926 20:10:54.120618 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.120625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.120696 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.120760 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.122872 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.123707 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.125952 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.126310 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.126386 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.133017 (+ 6631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.133022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.133025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.133059 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.135687 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.136223 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.136228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.136230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.136326 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.136642 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.136646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.136648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.136878 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.136982 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.137368 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.137494 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:10:54.137647 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.117753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:57793 (local address 127.23.50.190:41147)
0926 20:10:54.117988 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.117995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.119718 (+ 1723us) server_negotiation.cc:407] Connection header received
0926 20:10:54.120585 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.120588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.120646 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.120730 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.122874 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.123708 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.125943 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.126304 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.126386 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.130430 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.130433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.130436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.130466 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.132529 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.133298 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.133305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.133309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.133414 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.133912 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.133917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.133919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.134228 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.134348 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.137324 (+ 2976us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.137477 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 20:10:54.137717 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.120829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:48731 (local address 127.23.50.188:45035)
0926 20:10:54.121130 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.121135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.121575 (+ 440us) server_negotiation.cc:407] Connection header received
0926 20:10:54.121751 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.121757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.121827 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.121945 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.122868 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.124243 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.125946 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.126458 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.126584 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.131799 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.131802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.131805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.131837 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.133797 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.134823 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.134830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.134834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.134934 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.135292 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.135299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.135303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.135568 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.135674 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.137303 (+ 1629us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.137466 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":98,"threads_started":1}
I20260926 20:10:54.137142 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.119354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:45035 (local address 127.23.50.129:48731)
0926 20:10:54.121483 (+ 2129us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.121502 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.121615 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.122112 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.122118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.122366 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:54.122678 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.122702 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.124399 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.124406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.125763 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.125778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.126759 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.126767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.127012 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.127846 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:54.127877 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:54.131601 (+ 3724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:54.134047 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.134053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.134058 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.134694 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.135050 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.135053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.135057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.135160 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.135727 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:54.135733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:54.135853 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.135976 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.136160 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2028,"mutex_wait_us":121,"spinlock_wait_cycles":43776,"thread_start_us":80,"threads_started":1}
I20260926 20:10:54.138336 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.118234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:40407 (local address 127.23.50.129:34757)
0926 20:10:54.118847 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.118911 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.120365 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.120935 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.120941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.121727 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:54.122678 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.122701 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.123941 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.123944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.125766 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.125777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.126607 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.126611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.126962 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.127912 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:54.127932 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:54.132782 (+ 4850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:54.135824 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.135827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.135830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.136082 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.136444 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.136447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.136449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.136534 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.137057 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:54.137064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:54.137170 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.137917 (+ 747us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.138119 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":509,"spinlock_wait_cycles":14208,"thread_start_us":136,"threads_started":1}
I20260926 20:10:54.140512 30867 heartbeater.cc:347] Connected to a master server at 127.23.50.188:45035
I20260926 20:10:54.140555 30868 heartbeater.cc:347] Connected to a master server at 127.23.50.190:41147
I20260926 20:10:54.140810 30868 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:54.140884 30869 heartbeater.cc:347] Connected to a master server at 127.23.50.189:40407
I20260926 20:10:54.140957 30869 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:54.141465 30869 heartbeater.cc:511] Master 127.23.50.189:40407 requested a full tablet report, sending...
I20260926 20:10:54.141463 30868 heartbeater.cc:511] Master 127.23.50.190:41147 requested a full tablet report, sending...
I20260926 20:10:54.140825 30867 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:54.142053 30867 heartbeater.cc:511] Master 127.23.50.188:45035 requested a full tablet report, sending...
I20260926 20:10:54.142869 29902 ts_manager.cc:194] Re-registered known tserver with Master: 0c3b0b17121f483789d7ea584b9a5a79 (127.23.50.129:36633)
I20260926 20:10:54.143160 30688 ts_manager.cc:194] Registered new tserver with Master: 0c3b0b17121f483789d7ea584b9a5a79 (127.23.50.129:36633)
I20260926 20:10:54.143214 30496 ts_manager.cc:194] Registered new tserver with Master: 0c3b0b17121f483789d7ea584b9a5a79 (127.23.50.129:36633)
I20260926 20:10:54.143621 29902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:57793
I20260926 20:10:54.245363 30361 heartbeater.cc:503] Master 127.23.50.190:41147 was elected leader, sending a full tablet report...
W20260926 20:10:54.253751 30875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:54.254077 30875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:54.254122 30875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:54.258522 30875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:54.258622 30875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:54.258687 30875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:10:54.264717 30875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab
--principal=oryx/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:37165
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=34051
--tserver_master_addrs=127.23.50.190:41147,127.23.50.189:40407,127.23.50.188:45035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:54.266135 30875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:54.267254 30875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:54.277794 30883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:54.277948 30880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:54.278048 30881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:54.278088 30875 server_base.cc:1034] running on GCE node
I20260926 20:10:54.278530 30875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:54.279194 30875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:54.280359 30875 hybrid_clock.cc:648] HybridClock initialized: now 1790453454280342 us; error 28 us; skew 500 ppm
Sep 26 20:10:54 dist-test-slave-7034 krb5kdc[29857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453454, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:54.283330 30875 init.cc:377] Logged in from keytab as oryx/127.23.50.130@KRBTEST.COM (short username oryx)
I20260926 20:10:54.284250 30875 webserver.cc:460] Webserver started at http://127.23.50.130:34051/ using document root <none> and password file <none>
I20260926 20:10:54.284832 30875 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:54.284905 30875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:54.288954 30875 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 20:10:54.291397 30890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:54.292838 30875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:10:54.293000 30875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "c7758e9f4ea346a394c342bea4de2728"
format_stamp: "Formatted at 2026-09-26 20:10:52 on dist-test-slave-7034"
server_key: "349a457fbd0069d4b61a40390f5e2a66"
server_key_iv: "ad3919af6d70294998d85ec5059d3af1"
server_key_version: "encryptionkey@0"
I20260926 20:10:54.293605 30875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:54.305056 30875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:54.308965 30875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:54.309191 30875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:54.309913 30875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:54.310823 30875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:54.310869 30875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:54.310935 30875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:54.310966 30875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:54.319375 30875 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:37165
I20260926 20:10:54.319453 31003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:37165 every 8 connection(s)
I20260926 20:10:54.320448 30875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:10:54.328502 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 30875
I20260926 20:10:54.328682 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 30232
Sep 26 20:10:54 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453454, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.188@KRBTEST.COM
Sep 26 20:10:54 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453454, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
Sep 26 20:10:54 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453454, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.189@KRBTEST.COM
I20260926 20:10:54.337441 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:34899
--local_ip_for_outbound_sockets=127.23.50.131
--tserver_master_addrs=127.23.50.190:41147
--webserver_port=43139
--webserver_interface=127.23.50.131
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab
--principal=oryx/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.50.190:41147,127.23.50.189:40407
--tserver_master_addrs=127.23.50.190:41147,127.23.50.189:40407,127.23.50.188:45035 with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:10:54.337920 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.322405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:43135 (local address 127.23.50.190:41147)
0926 20:10:54.322553 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.322557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.323442 (+ 885us) server_negotiation.cc:407] Connection header received
0926 20:10:54.324627 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.324632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.324688 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.324759 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.326231 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.327044 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.328405 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.328739 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.328824 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.333239 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.333246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.333252 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.333289 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.335177 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.335838 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.335842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.335845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.335946 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.336421 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.336425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.336427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.336675 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.336758 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.337549 (+ 791us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.337684 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:10:54.338405 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.322478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:42921 (local address 127.23.50.189:40407)
0926 20:10:54.322630 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.322636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.323679 (+ 1043us) server_negotiation.cc:407] Connection header received
0926 20:10:54.324662 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.324665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.324740 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.324795 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.326209 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.326742 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.328114 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.328419 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.328531 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.334389 (+ 5858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.334395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.334401 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.334437 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.336343 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.336877 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.336884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.336888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.336986 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.337292 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.337298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.337303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.337532 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.337617 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.338046 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.338183 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:10:54.338794 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.324017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:49883 (local address 127.23.50.188:45035)
0926 20:10:54.324143 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.324146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.324160 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:10:54.324663 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.324666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.324723 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.324791 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.326234 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.327053 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.328405 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.328760 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.328840 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.333268 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.333271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.333274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.333306 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.336918 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.337393 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.337396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.337398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.337493 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.337788 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.337792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.337794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.338028 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.338116 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.338493 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.338645 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:10:54.340245 31009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.322816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:40407 (local address 127.23.50.130:42921)
0926 20:10:54.323565 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.323578 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.324427 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.324977 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.324979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.325431 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:54.326065 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.326081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.326896 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.326901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.327961 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.327971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.330701 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.330708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.330855 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.331424 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:54.331443 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:54.334256 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:54.336480 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.336483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.336485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.336746 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.337100 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.337103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.337105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.337194 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.337692 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:54.337697 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:54.337805 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.338563 (+ 758us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.339033 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":122,"spinlock_wait_cycles":24704,"thread_start_us":154,"threads_started":1}
I20260926 20:10:54.341795 31006 heartbeater.cc:347] Connected to a master server at 127.23.50.189:40407
I20260926 20:10:54.340245 31008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.322822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.130:43135)
0926 20:10:54.323291 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.323345 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.324428 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.324946 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.324955 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.325425 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:54.326076 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.326083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.327192 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.327195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.328260 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.328279 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.329280 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.329283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.329367 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.330181 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:54.330206 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:54.333088 (+ 2882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:54.335310 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.335317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.335333 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.335695 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.336070 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.336077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.336082 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.336293 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.336870 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:54.336878 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:54.337235 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.338719 (+ 1484us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.339145 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":454,"spinlock_wait_cycles":11392,"thread_start_us":113,"threads_started":1}
I20260926 20:10:54.342064 31006 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:54.342629 31006 heartbeater.cc:511] Master 127.23.50.189:40407 requested a full tablet report, sending...
I20260926 20:10:54.343040 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.329625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.23.50.188:45035)
0926 20:10:54.338974 (+ 9349us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.338978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.338990 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:10:54.339031 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.339034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.339090 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.339190 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:54.339673 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.340387 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.341761 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.342386 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.342497 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.342646 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.342849 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.342922 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9286,"thread_start_us":83,"threads_started":1}
I20260926 20:10:54.343125 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.329377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:45035 (local address 127.0.0.1:52934)
0926 20:10:54.329510 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.329526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.329603 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.339365 (+ 9762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.339372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.339394 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:54.339487 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.339494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.340528 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.340533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.341613 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.341624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.342529 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.342537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.342686 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.342729 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.342833 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.342941 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 20:10:54.343487 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.323713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:45035 (local address 127.23.50.130:49883)
0926 20:10:54.323992 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.324005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.324428 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.324946 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.324955 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.325462 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:54.326074 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.326083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.327192 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.327195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.328266 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.328271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.329025 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.329029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.329239 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.330181 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:54.330209 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:54.332649 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:54.337029 (+ 4380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.337032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.337034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.337257 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.337605 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.337608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.337610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.337689 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.338204 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:54.338208 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:54.338322 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.342684 (+ 4362us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.343299 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":229,"spinlock_wait_cycles":25856,"thread_start_us":103,"threads_started":1}
I20260926 20:10:54.343987 31004 heartbeater.cc:347] Connected to a master server at 127.23.50.188:45035
I20260926 20:10:54.344084 31004 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:54.344175 31005 heartbeater.cc:347] Connected to a master server at 127.23.50.190:41147
I20260926 20:10:54.344064 30698 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 20:10:54.344254 31005 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:54.344283 31004 heartbeater.cc:511] Master 127.23.50.188:45035 requested a full tablet report, sending...
I20260926 20:10:54.344424 31005 heartbeater.cc:511] Master 127.23.50.190:41147 requested a full tablet report, sending...
I20260926 20:10:54.344630 30727 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c0a1ed2d1614a04add0c82a6c22d24e. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:54.344803 30727 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: This master's current role is: LEARNER
I20260926 20:10:54.344875 30496 ts_manager.cc:194] Registered new tserver with Master: c7758e9f4ea346a394c342bea4de2728 (127.23.50.130:37165)
I20260926 20:10:54.345182 30688 ts_manager.cc:194] Registered new tserver with Master: c7758e9f4ea346a394c342bea4de2728 (127.23.50.130:37165)
I20260926 20:10:54.345904 29902 ts_manager.cc:194] Re-registered known tserver with Master: c7758e9f4ea346a394c342bea4de2728 (127.23.50.130:37165)
I20260926 20:10:54.346444 29902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:43135
W20260926 20:10:54.455778 31012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:54.456111 31012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:54.456209 31012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:54.460666 31012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:54.460755 31012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:54.460855 31012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:10:54.466603 31012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab
--principal=oryx/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:34899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=43139
--tserver_master_addrs=127.23.50.190:41147,127.23.50.189:40407,127.23.50.188:45035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:54.467840 31012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:54.469025 31012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:54.479094 31017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:54.479223 31012 server_base.cc:1034] running on GCE node
W20260926 20:10:54.479106 31018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:54.479166 31020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:54.479856 31012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:54.480590 31012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:54.481832 31012 hybrid_clock.cc:648] HybridClock initialized: now 1790453454481794 us; error 38 us; skew 500 ppm
Sep 26 20:10:54 dist-test-slave-7034 krb5kdc[29857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453454, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:54.485004 31012 init.cc:377] Logged in from keytab as oryx/127.23.50.131@KRBTEST.COM (short username oryx)
I20260926 20:10:54.486167 31012 webserver.cc:460] Webserver started at http://127.23.50.131:43139/ using document root <none> and password file <none>
I20260926 20:10:54.486790 31012 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:54.486909 31012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:54.491130 31012 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:10:54.493588 31027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:54.495039 31012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:10:54.495235 31012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "28732512ad3746219c398c03e2602ae5"
format_stamp: "Formatted at 2026-09-26 20:10:53 on dist-test-slave-7034"
server_key: "e202be93524c23c1da47b2293e825ad9"
server_key_iv: "c32f15219bbb62a9cc79232e32e930ea"
server_key_version: "encryptionkey@0"
I20260926 20:10:54.495946 31012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:54.513784 31012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:54.516992 31012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:54.517225 31012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:54.517874 31012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:10:54.518744 31012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:10:54.518815 31012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:54.518887 31012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:10:54.518930 31012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:54.528067 31012 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:34899
I20260926 20:10:54.528086 31140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:34899 every 8 connection(s)
I20260926 20:10:54.529093 31012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:10:54.535172 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 31012
Sep 26 20:10:54 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453454, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.188@KRBTEST.COM
Sep 26 20:10:54 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453454, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.189@KRBTEST.COM
Sep 26 20:10:54 dist-test-slave-7034 krb5kdc[29857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453454, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
I20260926 20:10:54.547628 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.532520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:35885 (local address 127.23.50.188:45035)
0926 20:10:54.532649 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.532653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.532761 (+ 108us) server_negotiation.cc:407] Connection header received
0926 20:10:54.533317 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.533321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.533394 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.533473 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.535796 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.536666 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.538339 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.538604 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.538734 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.542999 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.543007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.543011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.543046 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.544966 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.545650 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.545653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.545655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.545742 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.546197 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.546200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.546202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.546447 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.546541 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.547301 (+ 760us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.547429 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:10:54.548408 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.531017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:35377 (local address 127.23.50.190:41147)
0926 20:10:54.531139 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.531144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.532459 (+ 1315us) server_negotiation.cc:407] Connection header received
0926 20:10:54.533473 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.533483 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.533552 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.533645 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.535485 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.536362 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.539711 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.540108 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.540222 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.544288 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.544294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.544299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.544332 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.546089 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.546828 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.546832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.546833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.546923 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.547222 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.547228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.547233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.547460 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.547550 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.548145 (+ 595us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.548262 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:10:54.548877 31145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.531376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:40407 (local address 127.23.50.131:60155)
0926 20:10:54.531984 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.532052 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.533013 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.533688 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.533698 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.534443 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:54.535558 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.535567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.536581 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.536583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.538765 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.538777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.539643 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.539650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.539741 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.540229 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:54.540242 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:54.542336 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:54.544998 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.545004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.545025 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.545518 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.545878 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.545881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.545883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.546070 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.546608 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:54.546616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:54.547007 (+ 391us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.547226 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.547614 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":1513,"spinlock_wait_cycles":78208,"thread_start_us":707,"threads_started":1}
I20260926 20:10:54.549145 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.531376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.131:35377)
0926 20:10:54.532366 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.532382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.533027 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.533688 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.533698 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.534443 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:54.535290 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.535313 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.536550 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.536553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.539564 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.539578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.540385 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.540391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.540488 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.541250 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:54.541264 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:54.544100 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:54.546199 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.546201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.546203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.546713 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.547026 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.547029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.547030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.547114 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.547593 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:54.547596 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:54.547687 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.547778 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.547935 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":1728,"spinlock_wait_cycles":75136,"thread_start_us":156,"threads_started":1}
I20260926 20:10:54.549407 31147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.532330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:45035 (local address 127.23.50.131:35885)
0926 20:10:54.532668 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.532681 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.533053 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.533694 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.533698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.534435 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:10:54.535604 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.535618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.536832 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.536836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.538230 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.538241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.538775 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.538779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.538991 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.539680 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:10:54.539704 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:10:54.542828 (+ 3124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:10:54.545102 (+ 2274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.545108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.545113 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.545523 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.545904 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:10:54.545909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:10:54.545911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:10:54.546080 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:10:54.546614 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:10:54.546617 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:10:54.547009 (+ 392us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.547849 (+ 840us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.547993 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":515,"spinlock_wait_cycles":35328,"thread_start_us":82,"threads_started":1}
I20260926 20:10:54.550141 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.530969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:60155 (local address 127.23.50.189:40407)
0926 20:10:54.531117 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.531122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.532208 (+ 1086us) server_negotiation.cc:407] Connection header received
0926 20:10:54.533284 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.533291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.533351 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.533411 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.535691 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.536450 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.538917 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.539214 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.539353 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.542525 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.542530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.542535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.542568 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.544851 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.545651 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.545655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.545657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.545746 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.546195 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.546198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.546200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.546452 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.546549 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.549723 (+ 3174us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.549921 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 20:10:54.550437 31142 heartbeater.cc:347] Connected to a master server at 127.23.50.190:41147
I20260926 20:10:54.550534 31141 heartbeater.cc:347] Connected to a master server at 127.23.50.188:45035
I20260926 20:10:54.550698 31141 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:54.550747 31143 heartbeater.cc:347] Connected to a master server at 127.23.50.189:40407
I20260926 20:10:54.550872 31143 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:54.550699 31142 heartbeater.cc:464] Registering TS with master...
I20260926 20:10:54.551393 31141 heartbeater.cc:511] Master 127.23.50.188:45035 requested a full tablet report, sending...
I20260926 20:10:54.551391 31142 heartbeater.cc:511] Master 127.23.50.190:41147 requested a full tablet report, sending...
I20260926 20:10:54.551391 31143 heartbeater.cc:511] Master 127.23.50.189:40407 requested a full tablet report, sending...
I20260926 20:10:54.552704 29902 ts_manager.cc:194] Re-registered known tserver with Master: 28732512ad3746219c398c03e2602ae5 (127.23.50.131:34899)
I20260926 20:10:54.552778 30688 ts_manager.cc:194] Registered new tserver with Master: 28732512ad3746219c398c03e2602ae5 (127.23.50.131:34899)
I20260926 20:10:54.553435 30496 ts_manager.cc:194] Registered new tserver with Master: 28732512ad3746219c398c03e2602ae5 (127.23.50.131:34899)
I20260926 20:10:54.554250 29902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:35377
I20260926 20:10:54.557411 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.541006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.23.50.190:41147)
0926 20:10:54.541279 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.541284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.542502 (+ 1218us) server_negotiation.cc:407] Connection header received
0926 20:10:54.542626 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.542630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.542690 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.542782 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.545181 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.546100 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.547487 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.548166 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.548275 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.551471 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.551476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.551478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.551508 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.555430 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.555951 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.555956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.555959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.556057 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.556354 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.556358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.556361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.556608 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.556744 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.557053 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.557201 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":119,"threads_started":1}
I20260926 20:10:54.558000 31152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.537391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.23.50.189:40407)
0926 20:10:54.537657 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.537662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.537881 (+ 219us) server_negotiation.cc:407] Connection header received
0926 20:10:54.537997 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.538000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.538057 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.538170 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.539085 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.539922 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.541440 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.542480 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.542979 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.544370 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.544375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.544378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.544409 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.555879 (+ 11470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.556381 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.556384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.556386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.556466 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.556835 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.556838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.556839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.557054 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.557385 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.557668 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.557815 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
I20260926 20:10:54.562105 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.542404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.23.50.188:45035)
0926 20:10:54.543203 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.543210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.552091 (+ 8881us) server_negotiation.cc:407] Connection header received
0926 20:10:54.552251 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.552254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.552310 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.552396 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.553507 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.554375 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.555592 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.555951 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.556035 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.558464 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.558471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.558476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.558502 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.560449 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.560898 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.560902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.560904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.560990 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.561245 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.561248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.561250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.561456 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.561585 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.561806 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.561954 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705}
I20260926 20:10:54.560382 29902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:10:54.564016 29902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:54.566881 30465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: NON_VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: true } } }
I20260926 20:10:54.567047 30465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:54.570636 31164 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e: attempting to promote NON_VOTER e0dca02309bf476d8f5a04655fc1bf13 to VOTER
I20260926 20:10:54.570977 31164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } }
I20260926 20:10:54.572404 30506 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 FOLLOWER]: Refusing update from remote peer 8c0a1ed2d1614a04add0c82a6c22d24e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 20:10:54.572759 30465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [LEADER]: Connected to new peer: Peer: permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 20:10:54.574570 30698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 LEARNER]: Refusing update from remote peer 8c0a1ed2d1614a04add0c82a6c22d24e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 20:10:54.575023 31159 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 20:10:54.575327 31159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e0dca02309bf476d8f5a04655fc1bf13 (127.23.50.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } } }
I20260926 20:10:54.576656 31159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } } }
I20260926 20:10:54.576768 31159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:54.577486 31159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } } }
I20260926 20:10:54.577631 31159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:54.577942 30506 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e0dca02309bf476d8f5a04655fc1bf13 (127.23.50.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } } }
I20260926 20:10:54.578378 30698 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e0dca02309bf476d8f5a04655fc1bf13 (127.23.50.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } } }
I20260926 20:10:54.580181 30727 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } } }
I20260926 20:10:54.580313 30727 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:10:54.580857 31164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } } }
I20260926 20:10:54.580967 31164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:54.581197 31168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } } }
I20260926 20:10:54.581314 31168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:10:54.588819 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.582765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:37165 (local address 127.0.0.1:59404)
0926 20:10:54.583850 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.583864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.583944 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.584888 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.584891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.584909 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:54.585007 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.585012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.586346 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.586350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.587195 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.587204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.588506 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.588509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.588580 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.588602 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.588645 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.588699 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":107,"threads_started":1}
I20260926 20:10:54.588976 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.583402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.23.50.130:37165)
0926 20:10:54.584400 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.584410 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.584428 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:10:54.584488 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.584492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.584738 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.584893 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:54.585108 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.586233 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.587612 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.588387 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.588489 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.588642 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.588694 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.588790 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":107,"threads_started":1}
I20260926 20:10:54.589357 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.583126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36633 (local address 127.0.0.1:58682)
0926 20:10:54.584090 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.584099 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.584148 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.584646 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.584650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.584677 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:54.584775 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.584784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.586230 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.586234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.587963 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.587972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.588976 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.588980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.589078 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.589107 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.589162 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.589232 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":831,"thread_start_us":83,"threads_started":1}
I20260926 20:10:54.589986 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.583857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.23.50.129:36633)
0926 20:10:54.584172 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.584188 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.584210 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:10:54.584284 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.584289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.584500 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.584658 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:54.584912 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.586098 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.588084 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.588834 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.589536 (+ 702us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.589683 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.589734 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.589824 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
I20260926 20:10:54.590583 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.582445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:34899 (local address 127.0.0.1:44464)
0926 20:10:54.582575 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.582592 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.582671 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.583545 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.583550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.583568 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:54.583701 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.583709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.585463 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.585469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.586484 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.586494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.590222 (+ 3728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.590225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.590312 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.590333 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.590409 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.590479 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 20:10:54.590673 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.582650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.23.50.131:34899)
0926 20:10:54.582961 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.582976 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.583001 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:10:54.583089 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.583095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.583377 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.583570 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:54.584170 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.585359 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.586606 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.590097 (+ 3491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.590223 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.590357 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.590438 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.590530 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":124,"threads_started":1}
I20260926 20:10:54.592137 30938 tablet_service.cc:1467] Processing CreateTablet for tablet 3080010414054e12affa0721fae26f98 (DEFAULT_TABLE table=test-table [id=a3ade7d2ba6d4350a612b8f8c3dc203e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:54.592581 30801 tablet_service.cc:1467] Processing CreateTablet for tablet 3080010414054e12affa0721fae26f98 (DEFAULT_TABLE table=test-table [id=a3ade7d2ba6d4350a612b8f8c3dc203e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:54.593178 30938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3080010414054e12affa0721fae26f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:54.593518 30801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3080010414054e12affa0721fae26f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:54.597049 31075 tablet_service.cc:1467] Processing CreateTablet for tablet 3080010414054e12affa0721fae26f98 (DEFAULT_TABLE table=test-table [id=a3ade7d2ba6d4350a612b8f8c3dc203e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:54.598285 31075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3080010414054e12affa0721fae26f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:54.600768 31174 tablet_bootstrap.cc:492] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79: Bootstrap starting.
I20260926 20:10:54.603076 31174 tablet_bootstrap.cc:654] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:54.603830 31174 log.cc:824] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:54.606088 31174 tablet_bootstrap.cc:492] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79: No bootstrap required, opened a new log
I20260926 20:10:54.606199 31177 tablet_bootstrap.cc:492] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5: Bootstrap starting.
I20260926 20:10:54.606369 31174 ts_tablet_manager.cc:1397] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:10:54.606943 31175 tablet_bootstrap.cc:492] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728: Bootstrap starting.
I20260926 20:10:54.608603 31177 tablet_bootstrap.cc:654] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:54.609153 31175 tablet_bootstrap.cc:654] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:54.609498 31177 log.cc:824] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:54.610002 31175 log.cc:824] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:54.609977 31174 raft_consensus.cc:348] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } } peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } }
I20260926 20:10:54.610283 31174 raft_consensus.cc:374] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:54.610356 31174 raft_consensus.cc:729] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3b0b17121f483789d7ea584b9a5a79, State: Initialized, Role: FOLLOWER
I20260926 20:10:54.610869 31174 consensus_queue.cc:260] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } } peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } }
I20260926 20:10:54.611768 31177 tablet_bootstrap.cc:492] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5: No bootstrap required, opened a new log
I20260926 20:10:54.611893 31174 ts_tablet_manager.cc:1428] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:10:54.611953 31177 ts_tablet_manager.cc:1397] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260926 20:10:54.611991 31175 tablet_bootstrap.cc:492] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728: No bootstrap required, opened a new log
I20260926 20:10:54.612187 31175 ts_tablet_manager.cc:1397] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:10:54.612200 30868 heartbeater.cc:503] Master 127.23.50.190:41147 was elected leader, sending a full tablet report...
I20260926 20:10:54.616343 31175 raft_consensus.cc:348] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } } peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } }
I20260926 20:10:54.616636 31175 raft_consensus.cc:374] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:54.616717 31175 raft_consensus.cc:729] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7758e9f4ea346a394c342bea4de2728, State: Initialized, Role: FOLLOWER
I20260926 20:10:54.616511 31177 raft_consensus.cc:348] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } } peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } }
I20260926 20:10:54.616784 31177 raft_consensus.cc:374] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:54.616845 31177 raft_consensus.cc:729] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28732512ad3746219c398c03e2602ae5, State: Initialized, Role: FOLLOWER
I20260926 20:10:54.617184 31175 consensus_queue.cc:260] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } } peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } }
I20260926 20:10:54.617280 31177 consensus_queue.cc:260] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } } peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } }
W20260926 20:10:54.618212 30870 tablet.cc:2369] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:10:54.618535 31175 ts_tablet_manager.cc:1428] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:10:54.618923 31005 heartbeater.cc:503] Master 127.23.50.190:41147 was elected leader, sending a full tablet report...
I20260926 20:10:54.620184 31177 ts_tablet_manager.cc:1428] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260926 20:10:54.620675 31142 heartbeater.cc:503] Master 127.23.50.190:41147 was elected leader, sending a full tablet report...
I20260926 20:10:54.684068 31181 raft_consensus.cc:482] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:54.684312 31181 raft_consensus.cc:504] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } } peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } }
I20260926 20:10:54.685406 31181 leader_election.cc:288] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28732512ad3746219c398c03e2602ae5 (127.23.50.131:34899), 0c3b0b17121f483789d7ea584b9a5a79 (127.23.50.129:36633)
I20260926 20:10:54.689571 31010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.685694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:34899 (local address 127.23.50.130:59759)
0926 20:10:54.685896 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.685913 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.686046 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.686374 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.686380 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.686407 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:54.686546 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.686555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.687379 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.687382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.688288 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.688297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.689105 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.689109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.689222 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.689255 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.689330 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.689415 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 20:10:54.689596 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.685824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:59759 (local address 127.23.50.131:34899)
0926 20:10:54.686007 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.686012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.686030 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:10:54.686198 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.686203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.686259 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.686339 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:54.686659 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.687264 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.688433 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.688983 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.689086 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.689137 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.689364 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.689446 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 20:10:54.689729 31008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.686061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36633 (local address 127.23.50.130:54395)
0926 20:10:54.686179 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:54.686192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:54.686301 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:54.686576 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:54.686578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:54.686593 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:54.686724 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.686731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.687572 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.687575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.688534 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:54.688545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.689276 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.689279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:54.689413 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.689437 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:10:54.689521 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:10:54.689586 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 20:10:54.689774 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.686156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:54395 (local address 127.23.50.129:36633)
0926 20:10:54.686271 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.686276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.686296 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:10:54.686426 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.686429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.686472 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.686557 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:54.686852 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.687427 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.688683 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.689108 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.689278 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.689342 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.689557 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.689644 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:10:54.690500 30821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3080010414054e12affa0721fae26f98" candidate_uuid: "c7758e9f4ea346a394c342bea4de2728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3b0b17121f483789d7ea584b9a5a79" is_pre_election: true
I20260926 20:10:54.690508 31095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3080010414054e12affa0721fae26f98" candidate_uuid: "c7758e9f4ea346a394c342bea4de2728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28732512ad3746219c398c03e2602ae5" is_pre_election: true
I20260926 20:10:54.690851 31095 raft_consensus.cc:2454] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7758e9f4ea346a394c342bea4de2728 in term 0.
I20260926 20:10:54.690851 30821 raft_consensus.cc:2454] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7758e9f4ea346a394c342bea4de2728 in term 0.
I20260926 20:10:54.691371 30894 leader_election.cc:302] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28732512ad3746219c398c03e2602ae5, c7758e9f4ea346a394c342bea4de2728; no voters:
I20260926 20:10:54.691645 31181 raft_consensus.cc:2781] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:54.691741 31181 raft_consensus.cc:482] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:54.691800 31181 raft_consensus.cc:3037] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:54.692922 31181 raft_consensus.cc:504] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } } peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } }
I20260926 20:10:54.693298 31181 leader_election.cc:288] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [CANDIDATE]: Term 1 election: Requested vote from peers 28732512ad3746219c398c03e2602ae5 (127.23.50.131:34899), 0c3b0b17121f483789d7ea584b9a5a79 (127.23.50.129:36633)
I20260926 20:10:54.693650 31095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3080010414054e12affa0721fae26f98" candidate_uuid: "c7758e9f4ea346a394c342bea4de2728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28732512ad3746219c398c03e2602ae5"
I20260926 20:10:54.693753 30821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3080010414054e12affa0721fae26f98" candidate_uuid: "c7758e9f4ea346a394c342bea4de2728" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3b0b17121f483789d7ea584b9a5a79"
I20260926 20:10:54.693845 31095 raft_consensus.cc:3037] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:54.693913 30821 raft_consensus.cc:3037] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:54.694949 31095 raft_consensus.cc:2454] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7758e9f4ea346a394c342bea4de2728 in term 1.
I20260926 20:10:54.694964 30821 raft_consensus.cc:2454] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7758e9f4ea346a394c342bea4de2728 in term 1.
I20260926 20:10:54.695277 30894 leader_election.cc:302] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c3b0b17121f483789d7ea584b9a5a79, c7758e9f4ea346a394c342bea4de2728; no voters:
I20260926 20:10:54.695464 31181 raft_consensus.cc:2781] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:54.695729 31181 raft_consensus.cc:686] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 1 LEADER]: Becoming Leader. State: Replica: c7758e9f4ea346a394c342bea4de2728, State: Running, Role: LEADER
I20260926 20:10:54.696108 31181 consensus_queue.cc:237] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } } peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } }
I20260926 20:10:54.700300 29900 catalog_manager.cc:5495] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7758e9f4ea346a394c342bea4de2728 (127.23.50.130). New cstate: current_term: 1 leader_uuid: "c7758e9f4ea346a394c342bea4de2728" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } health_report { overall_health: UNKNOWN } } }
I20260926 20:10:54.742489 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.738709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59410 (local address 127.23.50.130:37165)
0926 20:10:54.738873 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.738877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.738892 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:10:54.738986 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.738989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.739045 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.739143 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:54.739451 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.740007 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.740859 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.741070 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.741150 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.741492 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:54.742025 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:54.742129 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.742199 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.742314 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 20:10:54.748615 30821 raft_consensus.cc:1261] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79 [term 1 FOLLOWER]: Refusing update from remote peer c7758e9f4ea346a394c342bea4de2728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:54.748631 31095 raft_consensus.cc:1261] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5 [term 1 FOLLOWER]: Refusing update from remote peer c7758e9f4ea346a394c342bea4de2728: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:54.749375 31181 consensus_queue.cc:1035] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:54.749696 31183 consensus_queue.cc:1035] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:54.759677 31188 mvcc.cc:205] Tried to move back new op lower bound from 7333697350642872320 to 7333697350438998016. Current Snapshot: MvccSnapshot[applied={T|T < 7333697350642872320}]
I20260926 20:10:54.761307 31186 mvcc.cc:205] Tried to move back new op lower bound from 7333697350642872320 to 7333697350438998016. Current Snapshot: MvccSnapshot[applied={T|T < 7333697350642872320}]
I20260926 20:10:54.770865 29900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:54.771106 29900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:54.775465 29900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c0a1ed2d1614a04add0c82a6c22d24e: Sending DeleteTablet for 3 replicas of tablet 3080010414054e12affa0721fae26f98
I20260926 20:10:54.776454 31075 tablet_service.cc:1514] Processing DeleteTablet for tablet 3080010414054e12affa0721fae26f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:54 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:44464
I20260926 20:10:54.776612 30801 tablet_service.cc:1514] Processing DeleteTablet for tablet 3080010414054e12affa0721fae26f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:54 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:58682
I20260926 20:10:54.777064 31195 tablet_replica.cc:330] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5: stopping tablet replica
I20260926 20:10:54.777501 31195 raft_consensus.cc:2229] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:54.777725 30938 tablet_service.cc:1514] Processing DeleteTablet for tablet 3080010414054e12affa0721fae26f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:54 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:59404
I20260926 20:10:54.777828 31196 tablet_replica.cc:330] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79: stopping tablet replica
I20260926 20:10:54.778124 31195 raft_consensus.cc:2258] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:54.778178 31196 raft_consensus.cc:2229] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:54.778488 31196 raft_consensus.cc:2258] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:54.780454 31195 ts_tablet_manager.cc:1905] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:54.780453 31196 ts_tablet_manager.cc:1905] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:54.780634 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 29870
I20260926 20:10:54.784092 31196 ts_tablet_manager.cc:1918] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:10:54.784219 31196 log.cc:1196] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-0/wal/wals/3080010414054e12affa0721fae26f98
I20260926 20:10:54.784621 31196 ts_tablet_manager.cc:1939] T 3080010414054e12affa0721fae26f98 P 0c3b0b17121f483789d7ea584b9a5a79: Deleting consensus metadata
I20260926 20:10:54.785048 31197 tablet_replica.cc:330] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728: stopping tablet replica
I20260926 20:10:54.785313 31195 ts_tablet_manager.cc:1918] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:10:54.785399 31197 raft_consensus.cc:2229] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:10:54.785508 31195 log.cc:1196] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/3080010414054e12affa0721fae26f98
I20260926 20:10:54.785960 31195 ts_tablet_manager.cc:1939] T 3080010414054e12affa0721fae26f98 P 28732512ad3746219c398c03e2602ae5: Deleting consensus metadata
W20260926 20:10:54.788414 30756 connection.cc:535] server connection from 127.0.0.1:58682 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:10:54.788761 31197 raft_consensus.cc:2258] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 20:10:54.789053 31031 connection.cc:535] server connection from 127.0.0.1:44464 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:10:54.790663 31197 ts_tablet_manager.cc:1905] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:54.793429 31197 ts_tablet_manager.cc:1918] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:10:54.793534 31197 log.cc:1196] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/3080010414054e12affa0721fae26f98
I20260926 20:10:54.793877 31197 ts_tablet_manager.cc:1939] T 3080010414054e12affa0721fae26f98 P c7758e9f4ea346a394c342bea4de2728: Deleting consensus metadata
W20260926 20:10:54.795019 30891 connection.cc:441] server connection from 127.0.0.1:59404 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59404 (request call id 1) could send its response
I20260926 20:10:54.803134 30626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.791134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.23.50.189:40407)
0926 20:10:54.791267 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.791272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.791980 (+ 708us) server_negotiation.cc:407] Connection header received
0926 20:10:54.792199 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.792203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.792263 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.792320 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.794237 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.794924 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.796066 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.796352 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.796416 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.798620 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.798624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.798627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.798656 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.801069 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.801699 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.801704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.801706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.801792 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.802129 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.802132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.802134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.802386 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.802500 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.802822 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.802948 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59,"spinlock_wait_cycles":10496}
I20260926 20:10:54.803966 31152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.779270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.23.50.189:40407)
0926 20:10:54.779419 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.779425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.792161 (+ 12736us) server_negotiation.cc:407] Connection header received
0926 20:10:54.792312 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.792314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.792349 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.792390 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.793677 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.794615 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.795833 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.796223 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.796297 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.798638 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.798642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.798645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.798674 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.801177 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.802117 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.802121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.802123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.802224 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.802564 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.802569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.802571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.802818 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.802936 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.803631 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.803760 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"spinlock_wait_cycles":1920}
I20260926 20:10:54.806824 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.791732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.23.50.188:45035)
0926 20:10:54.791853 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.791858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.797028 (+ 5170us) server_negotiation.cc:407] Connection header received
0926 20:10:54.797295 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.797299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.797362 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.797423 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.798506 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.799304 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.800753 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.801156 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.801251 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.803036 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.803040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.803042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.803070 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.804896 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.805387 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.805390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.805392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.805481 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.805798 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.805801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.805803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.806088 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.806229 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.806528 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.806647 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:10:54.813387 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:54.778932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.23.50.188:45035)
0926 20:10:54.779049 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:54.779054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:54.805314 (+ 26260us) server_negotiation.cc:407] Connection header received
0926 20:10:54.805485 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:54.805487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:54.805536 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:54.805587 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:54.806439 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.807277 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.808196 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:54.808456 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:54.808565 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:54.809918 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:54.809922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:54.809924 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:54.809950 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:54.811426 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.812235 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.812239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.812241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.812329 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:54.812625 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:54.812628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:54.812629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:54.812817 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:54.812916 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:54.813173 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 20:10:54.813271 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:10:55.762421 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:55.761971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.130:50529)
0926 20:10:55.762204 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:55.762310 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
W20260926 20:10:55.762814 31005 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:41147 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
I20260926 20:10:55.763569 31207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:55.763176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.131:53519)
0926 20:10:55.763397 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:55.763482 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
W20260926 20:10:55.763897 31142 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:41147 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
I20260926 20:10:55.764231 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:55.763872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.129:59657)
0926 20:10:55.764074 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:55.764154 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
W20260926 20:10:55.764532 30868 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:41147 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
I20260926 20:10:56.276542 31209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c0a1ed2d1614a04add0c82a6c22d24e)
I20260926 20:10:56.276808 31209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } }
I20260926 20:10:56.276897 31210 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c0a1ed2d1614a04add0c82a6c22d24e)
I20260926 20:10:56.277097 31210 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } }
I20260926 20:10:56.277937 31209 leader_election.cc:288] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147), e0dca02309bf476d8f5a04655fc1bf13 (127.23.50.188:45035)
I20260926 20:10:56.278193 31210 leader_election.cc:288] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147), 873b624c07244e059d19b23dbc5b9b88 (127.23.50.189:40407)
I20260926 20:10:56.278652 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.278098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39052)
0926 20:10:56.278422 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.278529 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":67,"threads_started":1}
I20260926 20:10:56.278951 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.278337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39068)
0926 20:10:56.278803 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.278866 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":67,"threads_started":1}
W20260926 20:10:56.279007 30488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
W20260926 20:10:56.279446 30672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
I20260926 20:10:56.280156 31212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.279895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39082)
0926 20:10:56.280016 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.280073 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 20:10:56.280280 30488 leader_election.cc:334] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147): Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
I20260926 20:10:56.280587 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.280091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39084)
0926 20:10:56.280429 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.280498 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 20:10:56.280709 30672 leader_election.cc:334] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147): Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
I20260926 20:10:56.282804 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.278424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:45035 (local address 127.0.0.1:52966)
0926 20:10:56.278810 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.278827 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:56.278934 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:56.279367 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:56.279371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:56.279394 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:56.279508 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:56.279517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:56.280556 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:56.280559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:56.281530 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:56.281539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:56.282376 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:56.282379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:56.282505 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:56.282531 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 20:10:56.282602 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:10:56.282664 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":22,"thread_start_us":63,"threads_started":1}
I20260926 20:10:56.282846 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.278763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52966 (local address 127.23.50.188:45035)
0926 20:10:56.279100 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:56.279143 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:56.279159 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:10:56.279213 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:56.279220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:56.279273 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:56.279354 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:56.279648 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:56.280442 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:56.281661 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:56.282241 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:56.282349 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:56.282408 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:56.282642 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 20:10:56.282713 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":71,"threads_started":1}
I20260926 20:10:56.283320 30698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "873b624c07244e059d19b23dbc5b9b88" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0dca02309bf476d8f5a04655fc1bf13" is_pre_election: true
I20260926 20:10:56.283469 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.278455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:40407 (local address 127.0.0.1:45902)
0926 20:10:56.279186 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.279198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:56.279273 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:56.279688 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:56.279691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:56.279717 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:56.279824 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:56.279831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:56.280999 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:56.281002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:56.281993 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:56.282002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:56.282989 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:56.282998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:56.283110 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:56.283135 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:10:56.283228 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 20:10:56.283298 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":3,"thread_start_us":51,"threads_started":1}
I20260926 20:10:56.283558 30698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873b624c07244e059d19b23dbc5b9b88 in term 1.
I20260926 20:10:56.283578 31216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.278730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.23.50.189:40407)
0926 20:10:56.279235 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:56.279242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:56.279260 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:10:56.279398 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:56.279401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:56.279450 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:56.279919 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:56.280024 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:56.280877 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:56.282101 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:56.282709 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:56.282813 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:56.282868 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:56.283386 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 20:10:56.283455 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":439,"thread_start_us":88,"threads_started":1}
I20260926 20:10:56.283916 30488 leader_election.cc:302] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 873b624c07244e059d19b23dbc5b9b88, e0dca02309bf476d8f5a04655fc1bf13; no voters: 8c0a1ed2d1614a04add0c82a6c22d24e
I20260926 20:10:56.283987 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0dca02309bf476d8f5a04655fc1bf13" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "873b624c07244e059d19b23dbc5b9b88" is_pre_election: true
I20260926 20:10:56.284192 30506 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0dca02309bf476d8f5a04655fc1bf13 in term 1.
I20260926 20:10:56.284276 31209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 20:10:56.284329 31209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c0a1ed2d1614a04add0c82a6c22d24e)
I20260926 20:10:56.284386 31209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:10:56.284526 30675 leader_election.cc:302] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 873b624c07244e059d19b23dbc5b9b88, e0dca02309bf476d8f5a04655fc1bf13; no voters: 8c0a1ed2d1614a04add0c82a6c22d24e
I20260926 20:10:56.284763 31210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 20:10:56.284837 31210 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c0a1ed2d1614a04add0c82a6c22d24e)
I20260926 20:10:56.284883 31210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:10:56.285543 31209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } }
I20260926 20:10:56.285950 31209 leader_election.cc:288] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [CANDIDATE]: Term 2 election: Requested vote from peers 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147), e0dca02309bf476d8f5a04655fc1bf13 (127.23.50.188:45035)
I20260926 20:10:56.285954 31210 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } }
I20260926 20:10:56.286350 31210 leader_election.cc:288] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [CANDIDATE]: Term 2 election: Requested vote from peers 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147), 873b624c07244e059d19b23dbc5b9b88 (127.23.50.189:40407)
I20260926 20:10:56.286489 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.286070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39086)
0926 20:10:56.286362 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.286422 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260926 20:10:56.286666 30698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "873b624c07244e059d19b23dbc5b9b88" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0dca02309bf476d8f5a04655fc1bf13"
I20260926 20:10:56.286695 30506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0dca02309bf476d8f5a04655fc1bf13" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "873b624c07244e059d19b23dbc5b9b88"
I20260926 20:10:56.286821 30698 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 873b624c07244e059d19b23dbc5b9b88 in current term 2: Already voted for candidate e0dca02309bf476d8f5a04655fc1bf13 in this term.
I20260926 20:10:56.286864 30506 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e0dca02309bf476d8f5a04655fc1bf13 in current term 2: Already voted for candidate 873b624c07244e059d19b23dbc5b9b88 in this term.
I20260926 20:10:56.287221 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.286513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39090)
0926 20:10:56.287113 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.287163 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535}
I20260926 20:10:56.287398 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.286959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39106)
0926 20:10:56.287287 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.287342 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255}
W20260926 20:10:56.287537 30488 leader_election.cc:334] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147): Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
I20260926 20:10:56.287657 30488 leader_election.cc:302] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 873b624c07244e059d19b23dbc5b9b88; no voters: 8c0a1ed2d1614a04add0c82a6c22d24e, e0dca02309bf476d8f5a04655fc1bf13
I20260926 20:10:56.287788 31209 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 20:10:56.287794 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.287587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39118)
0926 20:10:56.287693 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.287740 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 20:10:56.287928 30672 leader_election.cc:334] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147): Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
I20260926 20:10:56.287999 30672 leader_election.cc:302] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0dca02309bf476d8f5a04655fc1bf13; no voters: 873b624c07244e059d19b23dbc5b9b88, 8c0a1ed2d1614a04add0c82a6c22d24e
I20260926 20:10:56.288161 31210 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 20:10:56.763962 31219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.763432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.130:37115)
0926 20:10:56.763763 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.763838 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":85,"threads_started":1}
I20260926 20:10:56.765272 31220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.764663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.131:35973)
0926 20:10:56.764927 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.765082 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260926 20:10:56.765712 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:56.765230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.129:33277)
0926 20:10:56.765575 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:56.765639 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":74,"threads_started":1}
I20260926 20:10:57.765517 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.764975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.130:52109)
0926 20:10:57.765266 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.765397 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260926 20:10:57.766451 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.765975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.131:47235)
0926 20:10:57.766289 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.766366 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":59,"threads_started":1}
I20260926 20:10:57.766789 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.766418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.129:39129)
0926 20:10:57.766625 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.766684 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260926 20:10:57.839267 31225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:57.839416 31225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } }
I20260926 20:10:57.839969 31225 leader_election.cc:288] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147), e0dca02309bf476d8f5a04655fc1bf13 (127.23.50.188:45035)
I20260926 20:10:57.840653 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.840200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39336)
0926 20:10:57.840454 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.840534 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":55,"threads_started":1}
I20260926 20:10:57.840833 30698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "873b624c07244e059d19b23dbc5b9b88" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0dca02309bf476d8f5a04655fc1bf13" is_pre_election: true
I20260926 20:10:57.841038 30698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 873b624c07244e059d19b23dbc5b9b88 in term 2.
I20260926 20:10:57.841490 30488 leader_election.cc:302] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 873b624c07244e059d19b23dbc5b9b88, e0dca02309bf476d8f5a04655fc1bf13; no voters:
I20260926 20:10:57.841584 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.841332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39340)
0926 20:10:57.841434 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.841489 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 20:10:57.841692 30488 leader_election.cc:334] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147): Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
I20260926 20:10:57.841737 31225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 20:10:57.841861 31225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:57.841938 31225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:10:57.842950 31225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } }
I20260926 20:10:57.843286 31225 leader_election.cc:288] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [CANDIDATE]: Term 3 election: Requested vote from peers 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147), e0dca02309bf476d8f5a04655fc1bf13 (127.23.50.188:45035)
I20260926 20:10:57.843765 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.843505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39344)
0926 20:10:57.843608 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.843659 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 20:10:57.843974 30698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "873b624c07244e059d19b23dbc5b9b88" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e0dca02309bf476d8f5a04655fc1bf13"
I20260926 20:10:57.844115 30698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:10:57.844520 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.844320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39348)
0926 20:10:57.844418 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.844465 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 20:10:57.844619 30488 leader_election.cc:334] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147): Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
I20260926 20:10:57.845237 30698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 873b624c07244e059d19b23dbc5b9b88 in term 3.
I20260926 20:10:57.845551 30488 leader_election.cc:302] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 873b624c07244e059d19b23dbc5b9b88, e0dca02309bf476d8f5a04655fc1bf13; no voters: 8c0a1ed2d1614a04add0c82a6c22d24e
I20260926 20:10:57.845708 31225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:10:57.845973 31225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [term 3 LEADER]: Becoming Leader. State: Replica: 873b624c07244e059d19b23dbc5b9b88, State: Running, Role: LEADER
I20260926 20:10:57.846239 31225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } }
I20260926 20:10:57.847404 31228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 873b624c07244e059d19b23dbc5b9b88. Latest consensus state: current_term: 3 leader_uuid: "873b624c07244e059d19b23dbc5b9b88" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } } }
I20260926 20:10:57.847555 31228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:57.847868 31230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:57.850152 31230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:57.850569 31230 catalog_manager.cc:1259] Loaded cluster ID: d249234698a84df49b235ad8f79ffee2
I20260926 20:10:57.850616 31230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:57.851184 31230 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:57.851584 31230 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: Loaded TSK: 0
I20260926 20:10:57.852490 31230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:10:57.881645 30496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:10:57.885118 30496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:10:57.888512 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.888181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39368)
0926 20:10:57.888349 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.888415 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 20:10:57.888491 30698 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [term 3 FOLLOWER]: Refusing update from remote peer 873b624c07244e059d19b23dbc5b9b88: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 20:10:57.889084 31225 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 20:10:57.889446 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.889186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39380)
0926 20:10:57.889315 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.889376 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:10:57.889572 30488 consensus_peers.cc:487] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 -> Peer 8c0a1ed2d1614a04add0c82a6c22d24e (127.23.50.190:41147): Couldn't send request to peer 8c0a1ed2d1614a04add0c82a6c22d24e. Status: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 20:10:57.890892 31232 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 873b624c07244e059d19b23dbc5b9b88. Latest consensus state: current_term: 3 leader_uuid: "873b624c07244e059d19b23dbc5b9b88" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } } }
I20260926 20:10:57.891042 31232 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:10:57.892334 31225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "873b624c07244e059d19b23dbc5b9b88" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } } }
I20260926 20:10:57.892468 31225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88 [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:57.892566 31232 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "873b624c07244e059d19b23dbc5b9b88" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c0a1ed2d1614a04add0c82a6c22d24e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41147 } } peers { permanent_uuid: "873b624c07244e059d19b23dbc5b9b88" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 40407 } attrs { promote: false } } peers { permanent_uuid: "e0dca02309bf476d8f5a04655fc1bf13" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 45035 } attrs { promote: false } } }
I20260926 20:10:57.892674 31232 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0dca02309bf476d8f5a04655fc1bf13 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:10:57.892859 31233 mvcc.cc:205] Tried to move back new op lower bound from 7333697363506130944 to 7333697363340849152. Current Snapshot: MvccSnapshot[applied={T|T < 7333697363506130944}]
I20260926 20:10:57.903836 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.899183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36633 (local address 127.0.0.1:58692)
0926 20:10:57.899451 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.899462 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:57.899562 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:57.900179 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:57.900185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:57.900208 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:57.900323 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:57.900330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:57.901400 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:57.901404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:57.902417 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:57.902426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:57.903407 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:57.903410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:57.903507 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:57.903531 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:10:57.903594 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 20:10:57.903679 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":23,"thread_start_us":116,"threads_started":1}
I20260926 20:10:57.904063 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.899790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.23.50.131:34899)
0926 20:10:57.900015 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:57.900018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:57.900030 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:10:57.900075 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:57.900077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:57.900144 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:57.900231 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:57.900570 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:57.901360 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:57.902796 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:57.903367 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:57.903470 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:57.903545 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:57.903846 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 20:10:57.903936 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":60,"threads_started":1}
I20260926 20:10:57.904244 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.899579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.23.50.129:36633)
0926 20:10:57.899894 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:57.899900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:57.899913 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:10:57.899968 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:57.899972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:57.900060 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:57.900145 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:57.900451 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:57.901270 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:57.902567 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:57.903189 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:57.903286 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:57.903350 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:57.904038 (+ 688us) server_negotiation.cc:299] Negotiation successful
0926 20:10:57.904114 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260926 20:10:57.904034 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.899528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:34899 (local address 127.0.0.1:44468)
0926 20:10:57.899782 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.899796 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:57.899912 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:57.900330 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:57.900333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:57.900348 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:57.900452 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:57.900458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:57.901458 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:57.901462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:57.902684 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:57.902698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:57.903605 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:57.903608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:57.903717 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:57.903735 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 20:10:57.903827 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:10:57.903896 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":24,"thread_start_us":106,"threads_started":1}
I20260926 20:10:57.904933 30801 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab61817d90c4e51b26123814277a4c8 (DEFAULT_TABLE table=test-table [id=028d821b3e23469a8935ee23d3b4ffeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:57.904954 31075 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab61817d90c4e51b26123814277a4c8 (DEFAULT_TABLE table=test-table [id=028d821b3e23469a8935ee23d3b4ffeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:57.905241 30801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab61817d90c4e51b26123814277a4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:57.905246 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.898985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:37165 (local address 127.0.0.1:59422)
0926 20:10:57.899137 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.899149 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:57.899268 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:57.901065 (+ 1797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:57.901068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:57.901084 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:57.901199 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:57.901206 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:57.902268 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:57.902271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:57.903396 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:57.903404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:57.904852 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:57.904856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:57.904936 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:57.904967 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 20:10:57.905049 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:10:57.905113 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 20:10:57.905256 31075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab61817d90c4e51b26123814277a4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:57.905306 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.899803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.23.50.130:37165)
0926 20:10:57.900368 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:57.900374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:57.900794 (+ 420us) server_negotiation.cc:407] Connection header received
0926 20:10:57.900851 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:57.900854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:57.900904 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:57.901053 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:57.901320 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:57.902137 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:57.903522 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:57.904719 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:57.904825 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:57.904886 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:57.905082 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 20:10:57.905172 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":90,"threads_started":1}
I20260926 20:10:57.905750 30938 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab61817d90c4e51b26123814277a4c8 (DEFAULT_TABLE table=test-table [id=028d821b3e23469a8935ee23d3b4ffeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:10:57.906028 30938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab61817d90c4e51b26123814277a4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:57.907918 31243 tablet_bootstrap.cc:492] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79: Bootstrap starting.
I20260926 20:10:57.908572 31245 tablet_bootstrap.cc:492] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728: Bootstrap starting.
I20260926 20:10:57.908910 31243 tablet_bootstrap.cc:654] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:57.909031 31244 tablet_bootstrap.cc:492] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5: Bootstrap starting.
I20260926 20:10:57.909466 31245 tablet_bootstrap.cc:654] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:57.909976 31244 tablet_bootstrap.cc:654] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:57.910821 31243 tablet_bootstrap.cc:492] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79: No bootstrap required, opened a new log
I20260926 20:10:57.910835 31245 tablet_bootstrap.cc:492] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728: No bootstrap required, opened a new log
I20260926 20:10:57.910918 31245 ts_tablet_manager.cc:1397] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 20:10:57.910919 31243 ts_tablet_manager.cc:1397] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:10:57.911216 31244 tablet_bootstrap.cc:492] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5: No bootstrap required, opened a new log
I20260926 20:10:57.911290 31244 ts_tablet_manager.cc:1397] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:10:57.911329 31243 raft_consensus.cc:348] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } } peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } }
I20260926 20:10:57.911348 31245 raft_consensus.cc:348] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } } peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } }
I20260926 20:10:57.911429 31243 raft_consensus.cc:374] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:57.911453 31243 raft_consensus.cc:729] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3b0b17121f483789d7ea584b9a5a79, State: Initialized, Role: FOLLOWER
I20260926 20:10:57.911450 31245 raft_consensus.cc:374] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:57.911484 31245 raft_consensus.cc:729] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7758e9f4ea346a394c342bea4de2728, State: Initialized, Role: FOLLOWER
I20260926 20:10:57.911589 31243 consensus_queue.cc:260] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } } peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } }
I20260926 20:10:57.911628 31245 consensus_queue.cc:260] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } } peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } }
I20260926 20:10:57.911695 31244 raft_consensus.cc:348] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } } peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } }
I20260926 20:10:57.911788 31244 raft_consensus.cc:374] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:57.911825 31244 raft_consensus.cc:729] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28732512ad3746219c398c03e2602ae5, State: Initialized, Role: FOLLOWER
I20260926 20:10:57.911962 31243 ts_tablet_manager.cc:1428] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:10:57.911958 31244 consensus_queue.cc:260] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } } peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } }
I20260926 20:10:57.912016 31245 ts_tablet_manager.cc:1428] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:10:57.912320 31244 ts_tablet_manager.cc:1428] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:10:57.913575 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.913087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.131:53217)
0926 20:10:57.913433 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.913496 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271}
I20260926 20:10:57.914150 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.913087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.130:38249)
0926 20:10:57.914002 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.914073 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":824}
I20260926 20:10:57.914845 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.913806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.129:51719)
0926 20:10:57.914718 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.914775 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":836}
I20260926 20:10:57.914973 30869 heartbeater.cc:503] Master 127.23.50.189:40407 was elected leader, sending a full tablet report...
I20260926 20:10:57.914987 31143 heartbeater.cc:503] Master 127.23.50.189:40407 was elected leader, sending a full tablet report...
I20260926 20:10:57.915781 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.915458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.131:46409)
0926 20:10:57.915664 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.915718 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260926 20:10:57.918018 31006 heartbeater.cc:503] Master 127.23.50.189:40407 was elected leader, sending a full tablet report...
I20260926 20:10:57.918134 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:57.917780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.130:41003)
0926 20:10:57.917955 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:57.918011 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260926 20:10:58.033129 31144 tablet.cc:2369] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:10:58.074718 31007 tablet.cc:2369] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:10:58.317229 31249 raft_consensus.cc:482] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:10:58.317409 31249 raft_consensus.cc:504] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } } peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } }
I20260926 20:10:58.318665 31249 leader_election.cc:288] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7758e9f4ea346a394c342bea4de2728 (127.23.50.130:37165), 28732512ad3746219c398c03e2602ae5 (127.23.50.131:34899)
I20260926 20:10:58.323235 31242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.319197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:55213 (local address 127.23.50.130:37165)
0926 20:10:58.319328 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:58.319333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:58.319545 (+ 212us) server_negotiation.cc:407] Connection header received
0926 20:10:58.319638 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:58.319641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:58.319686 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:58.319762 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:58.320055 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:58.320843 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:58.321992 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:58.322611 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:58.322749 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:58.322804 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:58.323036 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 20:10:58.323120 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:10:58.323243 31252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.319021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:37165 (local address 127.23.50.129:55213)
0926 20:10:58.319458 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:58.319469 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:58.319543 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:58.319807 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:58.319810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:58.319824 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:58.319932 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:58.319938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:58.320960 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:58.320962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:58.321865 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:58.321875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:58.322778 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:58.322784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:58.322895 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:58.322922 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:10:58.323002 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:10:58.323081 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":181,"threads_started":1}
I20260926 20:10:58.323887 30958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab61817d90c4e51b26123814277a4c8" candidate_uuid: "0c3b0b17121f483789d7ea584b9a5a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7758e9f4ea346a394c342bea4de2728" is_pre_election: true
I20260926 20:10:58.324110 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.318980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:34899 (local address 127.23.50.129:37873)
0926 20:10:58.319168 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:58.319183 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:10:58.319307 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:10:58.319629 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:10:58.319633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:10:58.319656 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:10:58.319807 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:58.319814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:58.321874 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:58.321878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:58.322662 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:10:58.322701 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:10:58.323687 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:10:58.323694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:10:58.323800 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:58.323822 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:10:58.323921 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 20:10:58.323995 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 20:10:58.324165 31241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.319038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:37873 (local address 127.23.50.131:34899)
0926 20:10:58.319181 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:58.319188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:58.319279 (+ 91us) server_negotiation.cc:407] Connection header received
0926 20:10:58.319444 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:58.319446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:58.319493 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:58.319600 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:10:58.321000 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:58.321701 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:58.322862 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:58.323459 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:58.323575 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:58.323624 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:58.323960 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 20:10:58.324046 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:10:58.324158 30958 raft_consensus.cc:2454] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c3b0b17121f483789d7ea584b9a5a79 in term 0.
I20260926 20:10:58.324528 31095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab61817d90c4e51b26123814277a4c8" candidate_uuid: "0c3b0b17121f483789d7ea584b9a5a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28732512ad3746219c398c03e2602ae5" is_pre_election: true
I20260926 20:10:58.324663 31095 raft_consensus.cc:2454] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c3b0b17121f483789d7ea584b9a5a79 in term 0.
I20260926 20:10:58.324654 30754 leader_election.cc:302] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c3b0b17121f483789d7ea584b9a5a79, c7758e9f4ea346a394c342bea4de2728; no voters:
I20260926 20:10:58.324988 31249 raft_consensus.cc:2781] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:10:58.325078 31249 raft_consensus.cc:482] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:10:58.325143 31249 raft_consensus.cc:3037] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:58.326349 31249 raft_consensus.cc:504] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } } peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } }
I20260926 20:10:58.326747 31249 leader_election.cc:288] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [CANDIDATE]: Term 1 election: Requested vote from peers c7758e9f4ea346a394c342bea4de2728 (127.23.50.130:37165), 28732512ad3746219c398c03e2602ae5 (127.23.50.131:34899)
I20260926 20:10:58.327111 30958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab61817d90c4e51b26123814277a4c8" candidate_uuid: "0c3b0b17121f483789d7ea584b9a5a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7758e9f4ea346a394c342bea4de2728"
I20260926 20:10:58.327122 31095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab61817d90c4e51b26123814277a4c8" candidate_uuid: "0c3b0b17121f483789d7ea584b9a5a79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28732512ad3746219c398c03e2602ae5"
I20260926 20:10:58.327235 30958 raft_consensus.cc:3037] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:58.327268 31095 raft_consensus.cc:3037] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:58.328189 31095 raft_consensus.cc:2454] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c3b0b17121f483789d7ea584b9a5a79 in term 1.
I20260926 20:10:58.328395 30958 raft_consensus.cc:2454] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c3b0b17121f483789d7ea584b9a5a79 in term 1.
I20260926 20:10:58.328567 30757 leader_election.cc:302] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c3b0b17121f483789d7ea584b9a5a79, 28732512ad3746219c398c03e2602ae5; no voters:
I20260926 20:10:58.328759 31249 raft_consensus.cc:2781] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:58.329010 31249 raft_consensus.cc:686] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 1 LEADER]: Becoming Leader. State: Replica: 0c3b0b17121f483789d7ea584b9a5a79, State: Running, Role: LEADER
I20260926 20:10:58.329300 31249 consensus_queue.cc:237] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } } peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } }
I20260926 20:10:58.329918 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.329605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.129:44541)
0926 20:10:58.329750 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:58.329807 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 20:10:58.332413 30504 catalog_manager.cc:5495] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c3b0b17121f483789d7ea584b9a5a79 (127.23.50.129). New cstate: current_term: 1 leader_uuid: "0c3b0b17121f483789d7ea584b9a5a79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c3b0b17121f483789d7ea584b9a5a79" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 } health_report { overall_health: UNKNOWN } } }
I20260926 20:10:58.359413 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.355609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.23.50.129:36633)
0926 20:10:58.355756 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:58.355760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:58.355773 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:10:58.355853 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:58.355855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:58.355910 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:58.356003 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:10:58.356284 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:58.356802 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:58.357897 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:58.358114 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:58.358265 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:58.358514 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:10:58.359020 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:10:58.359180 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:58.359248 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 20:10:58.359301 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:10:58.362542 31095 raft_consensus.cc:1261] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5 [term 1 FOLLOWER]: Refusing update from remote peer 0c3b0b17121f483789d7ea584b9a5a79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:58.362739 30958 raft_consensus.cc:1261] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728 [term 1 FOLLOWER]: Refusing update from remote peer 0c3b0b17121f483789d7ea584b9a5a79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:10:58.363125 31253 consensus_queue.cc:1035] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28732512ad3746219c398c03e2602ae5" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:58.363332 31249 consensus_queue.cc:1035] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7758e9f4ea346a394c342bea4de2728" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:10:58.365891 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.365131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.131:56457)
0926 20:10:58.365697 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:58.365766 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489}
I20260926 20:10:58.366556 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.366017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.130:47447)
0926 20:10:58.366424 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:58.366483 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337}
I20260926 20:10:58.368718 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.368141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.129:51257)
0926 20:10:58.368581 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:58.368636 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369}
I20260926 20:10:58.371007 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.370662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.130:58787)
0926 20:10:58.370862 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:58.370921 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 20:10:58.373060 31223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.372758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.131:45997)
0926 20:10:58.372926 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:58.372985 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 20:10:58.374101 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.373870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.23.50.129:53831)
0926 20:10:58.373970 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:58.374031 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 20:10:58.376780 30504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:58.376925 30504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:10:58.381253 30504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 873b624c07244e059d19b23dbc5b9b88: Sending DeleteTablet for 3 replicas of tablet 8ab61817d90c4e51b26123814277a4c8
I20260926 20:10:58.382230 31075 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab61817d90c4e51b26123814277a4c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:58 UTC) from {username='oryx', principal='oryx/127.23.50.189@KRBTEST.COM'} at 127.0.0.1:44468
I20260926 20:10:58.382444 30938 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab61817d90c4e51b26123814277a4c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:58 UTC) from {username='oryx', principal='oryx/127.23.50.189@KRBTEST.COM'} at 127.0.0.1:59422
I20260926 20:10:58.382470 30801 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab61817d90c4e51b26123814277a4c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:10:58 UTC) from {username='oryx', principal='oryx/127.23.50.189@KRBTEST.COM'} at 127.0.0.1:58692
I20260926 20:10:58.382800 31267 tablet_replica.cc:330] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728: stopping tablet replica
I20260926 20:10:58.382856 31268 tablet_replica.cc:330] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79: stopping tablet replica
I20260926 20:10:58.382977 31268 raft_consensus.cc:2229] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:10:58.382984 31267 raft_consensus.cc:2229] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:58.383131 31267 raft_consensus.cc:2258] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:58.383514 31268 raft_consensus.cc:2258] T 8ab61817d90c4e51b26123814277a4c8 P 0c3b0b17121f483789d7ea584b9a5a79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:58.383795 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 30649
I20260926 20:10:58.385172 31267 ts_tablet_manager.cc:1905] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:58.386130 31266 tablet_replica.cc:330] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5: stopping tablet replica
I20260926 20:10:58.386293 31266 raft_consensus.cc:2229] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:10:58.386443 31266 raft_consensus.cc:2258] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:10:58.387079 31266 ts_tablet_manager.cc:1905] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:10:58.388063 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.387748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39392)
0926 20:10:58.387927 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:58.387981 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 20:10:58.388453 31267 ts_tablet_manager.cc:1918] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:10:58.388533 31267 log.cc:1196] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/8ab61817d90c4e51b26123814277a4c8
I20260926 20:10:58.388852 31267 ts_tablet_manager.cc:1939] T 8ab61817d90c4e51b26123814277a4c8 P c7758e9f4ea346a394c342bea4de2728: Deleting consensus metadata
I20260926 20:10:58.388856 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.388603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41147 (local address 127.0.0.1:39406)
0926 20:10:58.388735 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:58.388781 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 20:10:58.389729 31266 ts_tablet_manager.cc:1918] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:10:58.389711 30488 catalog_manager.cc:4841] TS c7758e9f4ea346a394c342bea4de2728 (127.23.50.130:37165): tablet 8ab61817d90c4e51b26123814277a4c8 (table test-table [id=028d821b3e23469a8935ee23d3b4ffeb]) successfully deleted
I20260926 20:10:58.389813 31266 log.cc:1196] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/8ab61817d90c4e51b26123814277a4c8
I20260926 20:10:58.390095 31266 ts_tablet_manager.cc:1939] T 8ab61817d90c4e51b26123814277a4c8 P 28732512ad3746219c398c03e2602ae5: Deleting consensus metadata
I20260926 20:10:58.390733 30491 catalog_manager.cc:4841] TS 28732512ad3746219c398c03e2602ae5 (127.23.50.131:34899): tablet 8ab61817d90c4e51b26123814277a4c8 (table test-table [id=028d821b3e23469a8935ee23d3b4ffeb]) successfully deleted
I20260926 20:10:58.391860 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 30875
I20260926 20:10:58.392074 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:58.391837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36633 (local address 127.0.0.1:58702)
0926 20:10:58.391951 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 20:10:58.392001 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:10:58.393169 30491 catalog_manager.cc:4584] TS 0c3b0b17121f483789d7ea584b9a5a79 (127.23.50.129:36633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ab61817d90c4e51b26123814277a4c8: Network error: Client connection negotiation failed: client connection to 127.23.50.129:36633: connect: Connection refused (error 111)
I20260926 20:10:58.399314 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 31012
I20260926 20:10:58.405074 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 30366
I20260926 20:10:58.411429 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 30468
2026-09-26T20:10:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7361 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:10:58 dist-test-slave-7034 krb5kdc[31271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:10:58 dist-test-slave-7034 krb5kdc[31271](info): set up 2 sockets
Sep 26 20:10:58 dist-test-slave-7034 krb5kdc[31271](info): commencing operation
krb5kdc: starting...
W20260926 20:10:59.459064 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:10:59 dist-test-slave-7034 krb5kdc[31271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:10:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:10:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.190.keytab.
Entry for principal oryx/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.190.keytab.
I20260926 20:10:59.589445 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:39359
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.190.keytab
--principal=oryx/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:39359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:10:59.708308 31284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:10:59.708647 31284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:10:59.708859 31284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:10:59.713300 31284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:10:59.713413 31284 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 20:10:59.713446 31284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:10:59.713490 31284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:10:59.713550 31284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:10:59.713590 31284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:10:59.719215 31284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:39359
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.190.keytab
--principal=oryx/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:39359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:10:59.720364 31284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:10:59.721448 31284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:10:59.730367 31289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:59.730367 31290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:10:59.730418 31292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:10:59.730664 31284 server_base.cc:1034] running on GCE node
I20260926 20:10:59.731446 31284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:10:59.732585 31284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:10:59.733793 31284 hybrid_clock.cc:648] HybridClock initialized: now 1790453459733778 us; error 38 us; skew 500 ppm
Sep 26 20:10:59 dist-test-slave-7034 krb5kdc[31271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453459, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:10:59.736953 31284 init.cc:377] Logged in from keytab as oryx/127.23.50.190@KRBTEST.COM (short username oryx)
I20260926 20:10:59.738075 31284 webserver.cc:460] Webserver started at http://127.23.50.190:34901/ using document root <none> and password file <none>
I20260926 20:10:59.738687 31284 fs_manager.cc:359] Metadata directory not provided
I20260926 20:10:59.738782 31284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:10:59.739027 31284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:10:59.741125 31284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "e103592e8212445c9f2cbb95fafbbccf"
format_stamp: "Formatted at 2026-09-26 20:10:59 on dist-test-slave-7034"
server_key: "2b9d638ca824723240040a6f33898bb6"
server_key_iv: "687844acf51cb7c6452c53f1b34aeb88"
server_key_version: "encryptionkey@0"
I20260926 20:10:59.741701 31284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "e103592e8212445c9f2cbb95fafbbccf"
format_stamp: "Formatted at 2026-09-26 20:10:59 on dist-test-slave-7034"
server_key: "2b9d638ca824723240040a6f33898bb6"
server_key_iv: "687844acf51cb7c6452c53f1b34aeb88"
server_key_version: "encryptionkey@0"
I20260926 20:10:59.745674 31284 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:10:59.748340 31299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:10:59.749689 31284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:10:59.749926 31284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "e103592e8212445c9f2cbb95fafbbccf"
format_stamp: "Formatted at 2026-09-26 20:10:59 on dist-test-slave-7034"
server_key: "2b9d638ca824723240040a6f33898bb6"
server_key_iv: "687844acf51cb7c6452c53f1b34aeb88"
server_key_version: "encryptionkey@0"
I20260926 20:10:59.750065 31284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:10:59.765582 31284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:10:59.769021 31284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:10:59.769248 31284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:10:59.777777 31284 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:39359
I20260926 20:10:59.777792 31361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:39359 every 8 connection(s)
I20260926 20:10:59.778853 31284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:10:59.781695 31362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:10:59.786476 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 31284
I20260926 20:10:59.786639 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:10:59.786928 23754 external_mini_cluster.cc:1428] Setting key 01b749a6820e58186a2e204519a3a19c
I20260926 20:10:59.787668 31362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf: Bootstrap starting.
I20260926 20:10:59.790289 31362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf: Neither blocks nor log segments found. Creating new log.
I20260926 20:10:59.791107 31362 log.cc:824] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:10:59.793081 31362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf: No bootstrap required, opened a new log
Sep 26 20:10:59 dist-test-slave-7034 krb5kdc[31271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
I20260926 20:10:59.797294 31362 raft_consensus.cc:348] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e103592e8212445c9f2cbb95fafbbccf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39359 } }
I20260926 20:10:59.797633 31362 raft_consensus.cc:374] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:10:59.797746 31362 raft_consensus.cc:729] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e103592e8212445c9f2cbb95fafbbccf, State: Initialized, Role: FOLLOWER
I20260926 20:10:59.798354 31362 consensus_queue.cc:260] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e103592e8212445c9f2cbb95fafbbccf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39359 } }
I20260926 20:10:59.798491 31362 raft_consensus.cc:388] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:10:59.798581 31362 raft_consensus.cc:482] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:10:59.798702 31362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:10:59.799645 31362 raft_consensus.cc:504] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e103592e8212445c9f2cbb95fafbbccf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39359 } }
I20260926 20:10:59.800024 31362 leader_election.cc:302] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e103592e8212445c9f2cbb95fafbbccf; no voters:
I20260926 20:10:59.800417 31362 leader_election.cc:288] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:10:59.800498 31367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:10:59.800714 31367 raft_consensus.cc:686] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [term 1 LEADER]: Becoming Leader. State: Replica: e103592e8212445c9f2cbb95fafbbccf, State: Running, Role: LEADER
I20260926 20:10:59.801030 31367 consensus_queue.cc:237] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e103592e8212445c9f2cbb95fafbbccf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39359 } }
I20260926 20:10:59.801877 31362 sys_catalog.cc:564] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:10:59.802712 31369 sys_catalog.cc:455] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e103592e8212445c9f2cbb95fafbbccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e103592e8212445c9f2cbb95fafbbccf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39359 } } }
I20260926 20:10:59.802861 31369 sys_catalog.cc:458] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [sys.catalog]: This master's current role is: LEADER
I20260926 20:10:59.803064 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:10:59.788446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.23.50.190:39359)
0926 20:10:59.788931 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 20:10:59.788947 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:10:59.788982 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:10:59.789705 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:10:59.789725 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:10:59.790160 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:10:59.790508 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:10:59.791335 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:59.792339 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:59.793196 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:10:59.793678 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:10:59.793899 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:10:59.796843 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:10:59.796849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:10:59.796864 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:10:59.796897 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:10:59.799115 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:59.799633 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:59.799639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:59.799646 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:59.799819 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:10:59.800121 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:10:59.800125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:10:59.800127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:10:59.800675 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:10:59.800908 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 20:10:59.801694 (+ 786us) server_negotiation.cc:299] Negotiation successful
0926 20:10:59.802035 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":143,"threads_started":1}
I20260926 20:10:59.803212 31368 sys_catalog.cc:455] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e103592e8212445c9f2cbb95fafbbccf. Latest consensus state: current_term: 1 leader_uuid: "e103592e8212445c9f2cbb95fafbbccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e103592e8212445c9f2cbb95fafbbccf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 39359 } } }
I20260926 20:10:59.803308 31368 sys_catalog.cc:458] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf [sys.catalog]: This master's current role is: LEADER
W20260926 20:10:59.807123 31382 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:10:59.807217 31382 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:10:59.807317 31376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:10:59.808050 31376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:10:59.814028 31376 catalog_manager.cc:1347] Generated new cluster ID: 38048ddcfb7747b18d7afe791d8f6901
I20260926 20:10:59.814119 31376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:10:59.827579 31376 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:10:59.828688 31376 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:10:59.844684 31376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf: Generated new TSK 0
I20260926 20:10:59.845544 31376 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab.
Entry for principal oryx/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab.
I20260926 20:10:59.887497 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39359
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab
--principal=oryx/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:00.017740 31388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:00.018162 31388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:00.018255 31388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:00.022531 31388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:00.022604 31388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:00.022650 31388 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:11:00.022717 31388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:11:00.028432 31388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab
--principal=oryx/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.50.190:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:00.029670 31388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:00.031131 31388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:00.041795 31396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:00.041810 31393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:00.041795 31394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:00.042186 31388 server_base.cc:1034] running on GCE node
I20260926 20:11:00.042627 31388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:00.043337 31388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:00.044548 31388 hybrid_clock.cc:648] HybridClock initialized: now 1790453460044517 us; error 43 us; skew 500 ppm
Sep 26 20:11:00 dist-test-slave-7034 krb5kdc[31271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453460, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:00.047699 31388 init.cc:377] Logged in from keytab as oryx/127.23.50.129@KRBTEST.COM (short username oryx)
I20260926 20:11:00.048637 31388 webserver.cc:460] Webserver started at http://127.23.50.129:36675/ using document root <none> and password file <none>
I20260926 20:11:00.049258 31388 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:00.049345 31388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:00.049556 31388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:00.051856 31388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "9d299ab7cc8e4d33868dea282d0c2e4c"
format_stamp: "Formatted at 2026-09-26 20:11:00 on dist-test-slave-7034"
server_key: "b599ae7738f0ee2c0147d9eeb61a7b57"
server_key_iv: "8f2c1e777c6e2c30e50abf3be176c3cf"
server_key_version: "encryptionkey@0"
I20260926 20:11:00.052376 31388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "9d299ab7cc8e4d33868dea282d0c2e4c"
format_stamp: "Formatted at 2026-09-26 20:11:00 on dist-test-slave-7034"
server_key: "b599ae7738f0ee2c0147d9eeb61a7b57"
server_key_iv: "8f2c1e777c6e2c30e50abf3be176c3cf"
server_key_version: "encryptionkey@0"
I20260926 20:11:00.056447 31388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:11:00.059181 31403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:00.060572 31388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:11:00.060735 31388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "9d299ab7cc8e4d33868dea282d0c2e4c"
format_stamp: "Formatted at 2026-09-26 20:11:00 on dist-test-slave-7034"
server_key: "b599ae7738f0ee2c0147d9eeb61a7b57"
server_key_iv: "8f2c1e777c6e2c30e50abf3be176c3cf"
server_key_version: "encryptionkey@0"
I20260926 20:11:00.060877 31388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:00.093273 31388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:00.096876 31388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:00.097105 31388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:00.098706 31388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:00.098790 31388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:00.098866 31388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:00.098919 31388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:11:00 dist-test-slave-7034 krb5kdc[31271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453460, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
I20260926 20:11:00.110096 31388 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:44795
I20260926 20:11:00.110116 31518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:44795 every 8 connection(s)
I20260926 20:11:00.111030 31388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:11:00.113695 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.099496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:36205 (local address 127.23.50.190:39359)
0926 20:11:00.099649 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.099654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.100599 (+ 945us) server_negotiation.cc:407] Connection header received
0926 20:11:00.101672 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.101676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.101737 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.101813 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:00.103551 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.104439 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.106287 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.106594 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.106715 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.109725 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:00.109731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:00.109734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:00.109768 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:00.111520 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.112132 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.112135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.112137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.112230 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.112612 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.112614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.112615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.112802 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:00.112880 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.113419 (+ 539us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.113566 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:11:00.114598 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.099965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39359 (local address 127.23.50.129:36205)
0926 20:11:00.100422 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:00.100473 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:00.101415 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:00.102318 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:00.102324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:00.102708 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:00.103343 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.103367 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.104565 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.104571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.106142 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.106153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.106766 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.106770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.106954 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.107597 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:00.107613 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:00.109541 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:00.111664 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:00.111670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:00.111685 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:00.112013 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:00.112352 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:00.112355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:00.112356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:00.112513 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:00.112903 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:00.112909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:00.113176 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 20:11:00.113345 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 20:11:00.113625 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":128,"threads_started":1}
I20260926 20:11:00.115207 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 31388
I20260926 20:11:00.115352 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:11:00.115736 23754 external_mini_cluster.cc:1428] Setting key 9fb3845d12dac4062b6df3c49c30517d
I20260926 20:11:00.116133 31519 heartbeater.cc:347] Connected to a master server at 127.23.50.190:39359
I20260926 20:11:00.116293 31519 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:00.116896 31519 heartbeater.cc:511] Master 127.23.50.190:39359 requested a full tablet report, sending...
I20260926 20:11:00.118567 31316 ts_manager.cc:194] Registered new tserver with Master: 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795)
I20260926 20:11:00.119926 31316 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:36205
WARNING: no policy specified for oryx/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:11:00.126780 31525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.118574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39359 (local address 127.23.50.129:54705)
0926 20:11:00.119393 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:00.119410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:00.119529 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:00.119805 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:00.119809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:00.120066 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:00.120219 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.120228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.121196 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.121200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.121800 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.121809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.122328 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.122332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.122423 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.122961 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:00.122973 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:00.123322 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:00.125177 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:00.125182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:00.125184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:00.125399 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:00.125687 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:00.125691 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:00.125693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:00.125779 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:00.126179 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:00.126182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:00.126283 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 20:11:00.126414 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 20:11:00.126570 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":6,"thread_start_us":124,"threads_started":1}
I20260926 20:11:00.126770 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.118669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:54705 (local address 127.23.50.190:39359)
0926 20:11:00.118804 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.118811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.119506 (+ 695us) server_negotiation.cc:407] Connection header received
0926 20:11:00.119627 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.119630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.119674 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.119760 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:00.120361 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.121051 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.121969 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.122205 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.122319 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.123538 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:00.123541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:00.123544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:00.123567 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:00.125057 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.125516 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.125519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.125521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.125593 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.125896 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.125899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.125901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.126081 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:00.126190 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.126466 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.126576 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab.
Entry for principal oryx/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab.
I20260926 20:11:00.145390 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39359
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab
--principal=oryx/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:00.261471 31528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:00.261758 31528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:00.261844 31528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:00.265991 31528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:00.266057 31528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:00.266096 31528 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:11:00.266173 31528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:11:00.271596 31528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab
--principal=oryx/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.50.190:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:00.272774 31528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:00.273831 31528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:00.283922 31536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:00.284016 31534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:00.284000 31533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:00.284232 31528 server_base.cc:1034] running on GCE node
I20260926 20:11:00.284711 31528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:00.285388 31528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:00.286553 31528 hybrid_clock.cc:648] HybridClock initialized: now 1790453460286537 us; error 28 us; skew 500 ppm
Sep 26 20:11:00 dist-test-slave-7034 krb5kdc[31271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453460, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:00.289479 31528 init.cc:377] Logged in from keytab as oryx/127.23.50.130@KRBTEST.COM (short username oryx)
I20260926 20:11:00.290473 31528 webserver.cc:460] Webserver started at http://127.23.50.130:45529/ using document root <none> and password file <none>
I20260926 20:11:00.291039 31528 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:00.291102 31528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:00.291338 31528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:00.293408 31528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef"
format_stamp: "Formatted at 2026-09-26 20:11:00 on dist-test-slave-7034"
server_key: "202e1aad77d4f450669f12cc6ecd31cd"
server_key_iv: "b003de42a45a5f40324a2be25af98225"
server_key_version: "encryptionkey@0"
I20260926 20:11:00.294046 31528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef"
format_stamp: "Formatted at 2026-09-26 20:11:00 on dist-test-slave-7034"
server_key: "202e1aad77d4f450669f12cc6ecd31cd"
server_key_iv: "b003de42a45a5f40324a2be25af98225"
server_key_version: "encryptionkey@0"
I20260926 20:11:00.297796 31528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:11:00.300559 31543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:00.302048 31528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:11:00.302191 31528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef"
format_stamp: "Formatted at 2026-09-26 20:11:00 on dist-test-slave-7034"
server_key: "202e1aad77d4f450669f12cc6ecd31cd"
server_key_iv: "b003de42a45a5f40324a2be25af98225"
server_key_version: "encryptionkey@0"
I20260926 20:11:00.302311 31528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:00.330307 31528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:00.333703 31528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:00.333946 31528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:00.335551 31528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:00.335602 31528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:00.335680 31528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:00.335711 31528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:11:00 dist-test-slave-7034 krb5kdc[31271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453460, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
I20260926 20:11:00.348052 31528 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:33535
I20260926 20:11:00.348101 31658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:33535 every 8 connection(s)
I20260926 20:11:00.349339 31528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:11:00.350790 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.336454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:37031 (local address 127.23.50.190:39359)
0926 20:11:00.336638 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.336645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.337581 (+ 936us) server_negotiation.cc:407] Connection header received
0926 20:11:00.338764 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.338768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.338830 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.338943 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:00.340459 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.341333 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.342807 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.343092 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.343236 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.346550 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:00.346556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:00.346560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:00.346599 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:00.348495 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.349046 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.349050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.349052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.349161 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.349532 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.349535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.349537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.349736 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:00.349848 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.350483 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.350614 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 20:11:00.351580 31552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.336899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39359 (local address 127.23.50.130:37031)
0926 20:11:00.337399 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:00.337451 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:00.338480 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:00.339161 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:00.339171 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:00.339662 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:00.340295 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.340320 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.341448 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.341451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.342671 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.342689 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.343330 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.343337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.343505 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.344147 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:00.344175 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:00.346279 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:00.348614 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:00.348619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:00.348631 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:00.348901 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:00.349261 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:00.349264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:00.349265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:00.349433 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:00.349879 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:00.349888 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:00.350198 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 20:11:00.350421 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 20:11:00.350759 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":133,"threads_started":1}
I20260926 20:11:00.352442 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 31528
I20260926 20:11:00.352567 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:11:00.352859 23754 external_mini_cluster.cc:1428] Setting key 0a0430875dfede7a4cb538e644e71be7
I20260926 20:11:00.353185 31659 heartbeater.cc:347] Connected to a master server at 127.23.50.190:39359
I20260926 20:11:00.353377 31659 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:00.354070 31659 heartbeater.cc:511] Master 127.23.50.190:39359 requested a full tablet report, sending...
I20260926 20:11:00.355141 31315 ts_manager.cc:194] Registered new tserver with Master: dd62bd9a8cd74e3cb2f96b9243b5b2ef (127.23.50.130:33535)
I20260926 20:11:00.355890 31315 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:37031
WARNING: no policy specified for oryx/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:11:00.364084 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.356089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:54027 (local address 127.23.50.190:39359)
0926 20:11:00.356391 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.356395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.356407 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:11:00.356445 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.356448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.356494 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.356578 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:00.357208 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.357783 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.358743 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.359007 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.359127 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.360452 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:00.360456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:00.360459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:00.360486 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:00.362212 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.362701 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.362705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.362709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.362793 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.363120 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.363125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.363127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.363366 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:00.363490 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.363775 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.363897 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260926 20:11:00.364110 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.355823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39359 (local address 127.23.50.130:54027)
0926 20:11:00.356167 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:00.356187 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:00.356291 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:00.356622 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:00.356629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:00.356916 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:00.357069 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.357076 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.357954 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.357958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.358606 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.358615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.359152 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.359156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.359260 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.359908 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:00.359931 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:00.360294 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:00.362351 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:00.362356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:00.362359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:00.362578 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:00.362895 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:00.362900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:00.362902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:00.363022 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:00.363477 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:00.363480 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:00.363592 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 20:11:00.363714 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 20:11:00.363888 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab.
Entry for principal oryx/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab.
I20260926 20:11:00.386101 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:39359
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab
--principal=oryx/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:00.500998 31668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:00.501281 31668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:00.501328 31668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:00.505903 31668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:00.505975 31668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:00.506018 31668 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 20:11:00.506053 31668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:11:00.511806 31668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab
--principal=oryx/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.50.190:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:00.512851 31668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:00.513866 31668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:00.523785 31674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:00.523769 31676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:00.523818 31673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:00.523985 31668 server_base.cc:1034] running on GCE node
I20260926 20:11:00.524614 31668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:00.525285 31668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:00.526481 31668 hybrid_clock.cc:648] HybridClock initialized: now 1790453460526461 us; error 36 us; skew 500 ppm
Sep 26 20:11:00 dist-test-slave-7034 krb5kdc[31271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453460, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:00.529621 31668 init.cc:377] Logged in from keytab as oryx/127.23.50.131@KRBTEST.COM (short username oryx)
I20260926 20:11:00.530648 31668 webserver.cc:460] Webserver started at http://127.23.50.131:42653/ using document root <none> and password file <none>
I20260926 20:11:00.531226 31668 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:00.531287 31668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:00.531499 31668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:00.533581 31668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "8a6e1d9ca84c4e339b168d7042d5d08d"
format_stamp: "Formatted at 2026-09-26 20:11:00 on dist-test-slave-7034"
server_key: "64dea177a3deab42551b8c357d6de280"
server_key_iv: "980f55804b7947feeec79516da2c7fa7"
server_key_version: "encryptionkey@0"
I20260926 20:11:00.534204 31668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "8a6e1d9ca84c4e339b168d7042d5d08d"
format_stamp: "Formatted at 2026-09-26 20:11:00 on dist-test-slave-7034"
server_key: "64dea177a3deab42551b8c357d6de280"
server_key_iv: "980f55804b7947feeec79516da2c7fa7"
server_key_version: "encryptionkey@0"
I20260926 20:11:00.538133 31668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:11:00.540983 31683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:00.542478 31668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:11:00.542635 31668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "8a6e1d9ca84c4e339b168d7042d5d08d"
format_stamp: "Formatted at 2026-09-26 20:11:00 on dist-test-slave-7034"
server_key: "64dea177a3deab42551b8c357d6de280"
server_key_iv: "980f55804b7947feeec79516da2c7fa7"
server_key_version: "encryptionkey@0"
I20260926 20:11:00.542768 31668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:00.570665 31668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:00.574155 31668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:00.574383 31668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:00.575870 31668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:00.575948 31668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:00.576016 31668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:00.576059 31668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 20:11:00 dist-test-slave-7034 krb5kdc[31271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453460, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
I20260926 20:11:00.587214 31668 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:37463
I20260926 20:11:00.587308 31798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:37463 every 8 connection(s)
I20260926 20:11:00.588338 31668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:11:00.591759 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.577028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:44099 (local address 127.23.50.190:39359)
0926 20:11:00.577204 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.577209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.578340 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 20:11:00.579546 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.579550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.579610 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.579705 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:00.581515 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.582378 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.584287 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.584534 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.584628 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.587928 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:00.587934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:00.587940 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:00.587973 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:00.589652 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.590285 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.590288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.590289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.590371 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.590729 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.590731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.590733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.590914 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:00.590988 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.591528 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.591633 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:11:00.592564 31692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.577621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39359 (local address 127.23.50.131:44099)
0926 20:11:00.578150 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:00.578195 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:00.579296 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:00.579957 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:00.579968 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:00.580495 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:00.581312 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.581337 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.582536 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.582540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.584125 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.584136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.584719 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.584727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.584992 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.585824 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:00.585873 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:00.587720 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:00.589784 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:00.589790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:00.589807 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:00.590105 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:00.590478 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:00.590481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:00.590482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:00.590629 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:00.591018 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:00.591023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:00.591310 (+ 287us) client_negotiation.cc:769] Sending connection context
0926 20:11:00.591463 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 20:11:00.591726 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":157,"threads_started":1}
I20260926 20:11:00.593324 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 31668
I20260926 20:11:00.593487 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:11:00.593840 23754 external_mini_cluster.cc:1428] Setting key 4ef48b5d89f481687f31a61f5747c8aa
I20260926 20:11:00.594233 31799 heartbeater.cc:347] Connected to a master server at 127.23.50.190:39359
I20260926 20:11:00.594400 31799 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:00.595098 31799 heartbeater.cc:511] Master 127.23.50.190:39359 requested a full tablet report, sending...
I20260926 20:11:00.596262 31316 ts_manager.cc:194] Registered new tserver with Master: 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463)
I20260926 20:11:00.596858 31316 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:44099
I20260926 20:11:00.605100 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.597432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:55851 (local address 127.23.50.190:39359)
0926 20:11:00.597563 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.597568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.597805 (+ 237us) server_negotiation.cc:407] Connection header received
0926 20:11:00.597949 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.597952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.597999 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.598091 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:00.598481 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.599199 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.600114 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.600387 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.600474 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.601636 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:00.601639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:00.601641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:00.601665 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:00.603283 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.603789 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.603792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.603793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.603871 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.604205 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.604209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.604211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.604460 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:00.604548 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.604866 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.604982 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:11:00.605131 31805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.597342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39359 (local address 127.23.50.131:55851)
0926 20:11:00.597714 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:00.597726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:00.597808 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:00.598100 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:00.598102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:00.598284 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:00.598377 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.598382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.599344 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.599347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.599972 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.599981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.600542 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.600546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.600655 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.601132 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:00.601147 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:00.601502 (+ 355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:00.603390 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:00.603393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:00.603394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:00.603679 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:00.603982 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:00.603984 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:00.603986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:00.604071 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:00.604588 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:00.604591 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:00.604686 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 20:11:00.604790 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 20:11:00.604912 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":109,"threads_started":1}
I20260926 20:11:00.608229 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 20:11:00 dist-test-slave-7034 krb5kdc[31271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453460, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 20:11:00 dist-test-slave-7034 krb5kdc[31271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:00 dist-test-slave-7034 krb5kdc[31271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:11:00.632944 31365 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:00.626338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37798 (local address 127.23.50.190:39359)
0926 20:11:00.626477 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.626481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.626493 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:11:00.626536 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.626538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.626576 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.626638 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:00.627383 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.627918 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.628854 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.629081 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.629174 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.632761 (+ 3587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37798: BlockingRecv error: recv got EOF from 127.0.0.1:37798 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 20:11:00 dist-test-slave-7034 krb5kdc[31271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453460, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
I20260926 20:11:00.643277 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.634414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.23.50.190:39359)
0926 20:11:00.634558 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.634561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.634609 (+ 48us) server_negotiation.cc:407] Connection header received
0926 20:11:00.634752 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.634755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.634799 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.634874 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:00.635539 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.636061 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.636904 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.637151 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.637234 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.639797 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:00.639800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:00.639803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:00.639834 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:00.641298 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.641731 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.641733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.641734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.641811 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:00.642212 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:00.642216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:00.642217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:00.642500 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:00.642669 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.642960 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.643083 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:11:00.647729 31316 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:11:00.650918 31316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:11:00.667081 31823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.660383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:33535 (local address 127.0.0.1:33026)
0926 20:11:00.661032 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:00.661088 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:00.661220 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:00.662518 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:00.662520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:00.662537 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:00.662745 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.662756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.664128 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.664131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.665446 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.665457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.666488 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.666491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.666590 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.666655 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 20:11:00.666764 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 20:11:00.666905 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":130,"thread_start_us":106,"threads_started":1}
I20260926 20:11:00.667112 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.661379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.23.50.130:33535)
0926 20:11:00.662094 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.662105 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.662128 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:11:00.662197 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.662202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.662365 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.662640 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:00.662883 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.663981 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.665576 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.666340 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.666657 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.666803 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.666848 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.666962 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":101,"threads_started":1}
I20260926 20:11:00.667081 31821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.660202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:44795 (local address 127.0.0.1:35614)
0926 20:11:00.661031 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:00.661499 (+ 468us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:00.661615 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:00.662404 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:00.662410 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:00.662433 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:00.662585 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.662594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.664119 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.664122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.665583 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.665592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.666708 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.666711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.666817 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.666836 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:11:00.666916 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:11:00.666967 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":111,"thread_start_us":68,"threads_started":1}
I20260926 20:11:00.667194 31822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.660372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.23.50.129:44795)
0926 20:11:00.660727 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.660735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.661798 (+ 1063us) server_negotiation.cc:407] Connection header received
0926 20:11:00.661957 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.661963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.662210 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.662418 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:00.662747 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.663982 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.665713 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.666353 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.666482 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.666638 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.666958 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.667052 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":120,"threads_started":1}
I20260926 20:11:00.668711 31825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.661185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.23.50.131:37463)
0926 20:11:00.661982 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.661992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.662012 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:11:00.662074 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.662078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.662233 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.662396 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:00.662872 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.663607 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.665401 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.666300 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.666431 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.666655 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.668454 (+ 1799us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.668563 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":139,"threads_started":1}
I20260926 20:11:00.668902 31824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.660958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:37463 (local address 127.0.0.1:60392)
0926 20:11:00.661437 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:00.661450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:00.661536 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:00.662549 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:00.662551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:00.662566 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:00.662738 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.662744 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.663735 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.663740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.665266 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.665277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.666439 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.666446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.666574 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.667280 (+ 706us) client_negotiation.cc:769] Sending connection context
0926 20:11:00.667352 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:11:00.667435 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":146,"thread_start_us":89,"threads_started":1}
I20260926 20:11:00.670490 31453 tablet_service.cc:1467] Processing CreateTablet for tablet e485cce8e08e4c2e874c439b03f5059e (DEFAULT_TABLE table=test-table [id=52a329d26adb49c9a9a2223786ab5aa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:00.670491 31593 tablet_service.cc:1467] Processing CreateTablet for tablet e485cce8e08e4c2e874c439b03f5059e (DEFAULT_TABLE table=test-table [id=52a329d26adb49c9a9a2223786ab5aa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:00.671558 31453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e485cce8e08e4c2e874c439b03f5059e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:00.671646 31593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e485cce8e08e4c2e874c439b03f5059e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:00.672741 31733 tablet_service.cc:1467] Processing CreateTablet for tablet e485cce8e08e4c2e874c439b03f5059e (DEFAULT_TABLE table=test-table [id=52a329d26adb49c9a9a2223786ab5aa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:00.673831 31733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e485cce8e08e4c2e874c439b03f5059e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:00.677582 31827 tablet_bootstrap.cc:492] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c: Bootstrap starting.
I20260926 20:11:00.680018 31827 tablet_bootstrap.cc:654] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:00.680763 31828 tablet_bootstrap.cc:492] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Bootstrap starting.
I20260926 20:11:00.680919 31827 log.cc:824] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:11:00.682222 31830 tablet_bootstrap.cc:492] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d: Bootstrap starting.
I20260926 20:11:00.683065 31827 tablet_bootstrap.cc:492] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c: No bootstrap required, opened a new log
I20260926 20:11:00.683086 31828 tablet_bootstrap.cc:654] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:00.683300 31827 ts_tablet_manager.cc:1397] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:11:00.684155 31828 log.cc:824] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Log is configured to *not* fsync() on all Append() calls
I20260926 20:11:00.684479 31830 tablet_bootstrap.cc:654] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:00.685420 31830 log.cc:824] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:11:00.686316 31828 tablet_bootstrap.cc:492] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef: No bootstrap required, opened a new log
I20260926 20:11:00.686532 31828 ts_tablet_manager.cc:1397] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:11:00.687510 31830 tablet_bootstrap.cc:492] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d: No bootstrap required, opened a new log
I20260926 20:11:00.687703 31830 ts_tablet_manager.cc:1397] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:11:00.688004 31827 raft_consensus.cc:348] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.688323 31827 raft_consensus.cc:374] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:00.688385 31827 raft_consensus.cc:729] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d299ab7cc8e4d33868dea282d0c2e4c, State: Initialized, Role: FOLLOWER
I20260926 20:11:00.688870 31827 consensus_queue.cc:260] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.690001 31827 ts_tablet_manager.cc:1428] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260926 20:11:00.690070 31828 raft_consensus.cc:348] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.690343 31828 raft_consensus.cc:374] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:00.690397 31828 raft_consensus.cc:729] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd62bd9a8cd74e3cb2f96b9243b5b2ef, State: Initialized, Role: FOLLOWER
I20260926 20:11:00.690390 31519 heartbeater.cc:503] Master 127.23.50.190:39359 was elected leader, sending a full tablet report...
I20260926 20:11:00.690842 31828 consensus_queue.cc:260] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.691582 31659 heartbeater.cc:503] Master 127.23.50.190:39359 was elected leader, sending a full tablet report...
I20260926 20:11:00.691970 31828 ts_tablet_manager.cc:1428] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:11:00.692342 31830 raft_consensus.cc:348] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.692633 31830 raft_consensus.cc:374] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:00.692701 31830 raft_consensus.cc:729] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a6e1d9ca84c4e339b168d7042d5d08d, State: Initialized, Role: FOLLOWER
I20260926 20:11:00.693128 31830 consensus_queue.cc:260] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.693789 31799 heartbeater.cc:503] Master 127.23.50.190:39359 was elected leader, sending a full tablet report...
I20260926 20:11:00.694267 31830 ts_tablet_manager.cc:1428] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 20:11:00.758510 31834 raft_consensus.cc:482] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:11:00.758731 31834 raft_consensus.cc:504] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.759860 31834 leader_election.cc:288] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795), 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463)
I20260926 20:11:00.763777 31552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.760201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:37463 (local address 127.23.50.130:54739)
0926 20:11:00.760354 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:00.760372 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:00.760496 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:00.760780 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:00.760783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:00.760801 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:00.760923 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.760928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.761690 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.761695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.762624 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.762634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.763283 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.763286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.763458 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.763476 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 20:11:00.763566 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 20:11:00.763639 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":50}
I20260926 20:11:00.763777 31825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.760349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:54739 (local address 127.23.50.131:37463)
0926 20:11:00.760472 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.760476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.760492 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:11:00.760603 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.760606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.760655 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.760746 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:00.761045 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.761573 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.762760 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.763151 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.763290 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.763352 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.763583 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.763655 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:11:00.764576 31836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.760208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:44795 (local address 127.23.50.130:60887)
0926 20:11:00.760758 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:00.760770 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:00.760899 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:00.761166 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:00.761169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:00.761184 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:00.761696 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.761703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.762582 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.762588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.763377 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:00.763386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.764160 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.764166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:00.764265 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.764287 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:11:00.764368 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:11:00.764427 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":137,"thread_start_us":200,"threads_started":1}
I20260926 20:11:00.764508 31753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e485cce8e08e4c2e874c439b03f5059e" candidate_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" is_pre_election: true
I20260926 20:11:00.764652 31822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.760350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:60887 (local address 127.23.50.129:44795)
0926 20:11:00.760493 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.760499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.760859 (+ 360us) server_negotiation.cc:407] Connection header received
0926 20:11:00.761010 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.761013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.761061 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.761137 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:00.761815 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.762472 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.763502 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.764021 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.764140 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.764192 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.764402 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.764519 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:11:00.764847 31753 raft_consensus.cc:2454] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd62bd9a8cd74e3cb2f96b9243b5b2ef in term 0.
I20260926 20:11:00.765426 31546 leader_election.cc:302] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a6e1d9ca84c4e339b168d7042d5d08d, dd62bd9a8cd74e3cb2f96b9243b5b2ef; no voters:
I20260926 20:11:00.765491 31461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e485cce8e08e4c2e874c439b03f5059e" candidate_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" is_pre_election: true
I20260926 20:11:00.765761 31461 raft_consensus.cc:2454] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd62bd9a8cd74e3cb2f96b9243b5b2ef in term 0.
I20260926 20:11:00.765761 31834 raft_consensus.cc:2781] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:11:00.765865 31834 raft_consensus.cc:482] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:11:00.765920 31834 raft_consensus.cc:3037] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:00.767015 31834 raft_consensus.cc:504] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.767323 31834 leader_election.cc:288] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [CANDIDATE]: Term 1 election: Requested vote from peers 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795), 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463)
I20260926 20:11:00.767675 31461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e485cce8e08e4c2e874c439b03f5059e" candidate_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c"
I20260926 20:11:00.767714 31753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e485cce8e08e4c2e874c439b03f5059e" candidate_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d"
I20260926 20:11:00.767776 31461 raft_consensus.cc:3037] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:00.767866 31753 raft_consensus.cc:3037] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:00.768966 31753 raft_consensus.cc:2454] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd62bd9a8cd74e3cb2f96b9243b5b2ef in term 1.
I20260926 20:11:00.769001 31461 raft_consensus.cc:2454] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd62bd9a8cd74e3cb2f96b9243b5b2ef in term 1.
I20260926 20:11:00.769313 31546 leader_election.cc:302] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a6e1d9ca84c4e339b168d7042d5d08d, dd62bd9a8cd74e3cb2f96b9243b5b2ef; no voters:
I20260926 20:11:00.769512 31834 raft_consensus.cc:2781] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:11:00.769742 31834 raft_consensus.cc:686] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 LEADER]: Becoming Leader. State: Replica: dd62bd9a8cd74e3cb2f96b9243b5b2ef, State: Running, Role: LEADER
I20260926 20:11:00.770141 31834 consensus_queue.cc:237] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.773617 31315 catalog_manager.cc:5495] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef reported cstate change: term changed from 0 to 1, leader changed from <none> to dd62bd9a8cd74e3cb2f96b9243b5b2ef (127.23.50.130). New cstate: current_term: 1 leader_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } health_report { overall_health: UNKNOWN } } }
I20260926 20:11:00.798401 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:00.794602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.23.50.130:33535)
0926 20:11:00.794795 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:00.794802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:00.794819 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:11:00.794932 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:00.794935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:00.794998 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:00.795114 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:11:00.795428 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.795944 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.796819 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:00.797042 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:00.797158 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:00.797471 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:11:00.797990 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:11:00.798119 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:00.798190 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 20:11:00.798250 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 20:11:00.804525 31753 raft_consensus.cc:1261] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Refusing update from remote peer dd62bd9a8cd74e3cb2f96b9243b5b2ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:11:00.804525 31461 raft_consensus.cc:1261] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Refusing update from remote peer dd62bd9a8cd74e3cb2f96b9243b5b2ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:11:00.805248 31837 consensus_queue.cc:1035] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:11:00.805541 31834 consensus_queue.cc:1035] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:11:00.813423 31841 mvcc.cc:205] Tried to move back new op lower bound from 7333697375447891968 to 7333697375317610496. Current Snapshot: MvccSnapshot[applied={T|T < 7333697375447891968}]
I20260926 20:11:00.815825 31842 mvcc.cc:205] Tried to move back new op lower bound from 7333697375447891968 to 7333697375317610496. Current Snapshot: MvccSnapshot[applied={T|T < 7333697375447891968}]
I20260926 20:11:00.823076 31316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:11:00.823276 31316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:11:00.826006 31316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf: Sending DeleteTablet for 3 replicas of tablet e485cce8e08e4c2e874c439b03f5059e
I20260926 20:11:00.826937 31453 tablet_service.cc:1514] Processing DeleteTablet for tablet e485cce8e08e4c2e874c439b03f5059e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:00 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:35614
I20260926 20:11:00.827016 31593 tablet_service.cc:1514] Processing DeleteTablet for tablet e485cce8e08e4c2e874c439b03f5059e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:00 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:33026
I20260926 20:11:00.827173 31733 tablet_service.cc:1514] Processing DeleteTablet for tablet e485cce8e08e4c2e874c439b03f5059e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:00 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:60392
I20260926 20:11:00.827692 31849 tablet_replica.cc:330] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c: stopping tablet replica
I20260926 20:11:00.827723 31851 tablet_replica.cc:330] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d: stopping tablet replica
I20260926 20:11:00.828030 31849 raft_consensus.cc:2229] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:11:00.828259 31851 raft_consensus.cc:2229] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:11:00.828244 31316 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 20:11:00.828387 31849 raft_consensus.cc:2258] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 20:11:00.828737 31316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:11:00.828761 31850 tablet_replica.cc:330] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef: stopping tablet replica
I20260926 20:11:00.828938 31851 raft_consensus.cc:2258] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:00.829038 31850 raft_consensus.cc:2229] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:11:00.829432 31850 raft_consensus.cc:2258] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:00.830814 31851 ts_tablet_manager.cc:1905] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:00.830926 31850 ts_tablet_manager.cc:1905] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:00.833853 31849 ts_tablet_manager.cc:1905] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:00.834543 31453 tablet_service.cc:1467] Processing CreateTablet for tablet eca966abb4d8414ba94db205b347a5aa (DEFAULT_TABLE table=test-table [id=9b812acf614f4b4bad799623bd671c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:00.834872 31453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca966abb4d8414ba94db205b347a5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:00.835022 31850 ts_tablet_manager.cc:1918] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:11:00.835141 31850 log.cc:1196] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/e485cce8e08e4c2e874c439b03f5059e
I20260926 20:11:00.835510 31850 ts_tablet_manager.cc:1939] T e485cce8e08e4c2e874c439b03f5059e P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Deleting consensus metadata
I20260926 20:11:00.836583 31849 ts_tablet_manager.cc:1918] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:11:00.836706 31849 log.cc:1196] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/wal/wals/e485cce8e08e4c2e874c439b03f5059e
I20260926 20:11:00.837010 31849 ts_tablet_manager.cc:1939] T e485cce8e08e4c2e874c439b03f5059e P 9d299ab7cc8e4d33868dea282d0c2e4c: Deleting consensus metadata
I20260926 20:11:00.837514 31827 tablet_bootstrap.cc:492] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c: Bootstrap starting.
I20260926 20:11:00.838039 31593 tablet_service.cc:1467] Processing CreateTablet for tablet eca966abb4d8414ba94db205b347a5aa (DEFAULT_TABLE table=test-table [id=9b812acf614f4b4bad799623bd671c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:00.838057 31733 tablet_service.cc:1467] Processing CreateTablet for tablet eca966abb4d8414ba94db205b347a5aa (DEFAULT_TABLE table=test-table [id=9b812acf614f4b4bad799623bd671c6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:00.838306 31593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca966abb4d8414ba94db205b347a5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:00.838320 31733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca966abb4d8414ba94db205b347a5aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:00.838627 31827 tablet_bootstrap.cc:654] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:00.840232 31827 tablet_bootstrap.cc:492] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c: No bootstrap required, opened a new log
I20260926 20:11:00.840324 31827 ts_tablet_manager.cc:1397] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:11:00.840515 31301 catalog_manager.cc:4841] TS 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795): tablet e485cce8e08e4c2e874c439b03f5059e (table test-table [id=52a329d26adb49c9a9a2223786ab5aa3]) successfully deleted
I20260926 20:11:00.840714 31830 tablet_bootstrap.cc:492] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d: Bootstrap starting.
I20260926 20:11:00.840899 31828 tablet_bootstrap.cc:492] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Bootstrap starting.
I20260926 20:11:00.841223 31827 raft_consensus.cc:348] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.841349 31827 raft_consensus.cc:374] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:00.841392 31827 raft_consensus.cc:729] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d299ab7cc8e4d33868dea282d0c2e4c, State: Initialized, Role: FOLLOWER
I20260926 20:11:00.841526 31827 consensus_queue.cc:260] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.841627 31830 tablet_bootstrap.cc:654] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:00.841794 31827 ts_tablet_manager.cc:1428] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:11:00.841990 31302 catalog_manager.cc:4841] TS dd62bd9a8cd74e3cb2f96b9243b5b2ef (127.23.50.130:33535): tablet e485cce8e08e4c2e874c439b03f5059e (table test-table [id=52a329d26adb49c9a9a2223786ab5aa3]) successfully deleted
I20260926 20:11:00.842338 31828 tablet_bootstrap.cc:654] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:00.843628 31828 tablet_bootstrap.cc:492] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef: No bootstrap required, opened a new log
I20260926 20:11:00.843720 31828 ts_tablet_manager.cc:1397] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260926 20:11:00.844307 31828 raft_consensus.cc:348] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.844432 31828 raft_consensus.cc:374] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:00.844470 31828 raft_consensus.cc:729] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd62bd9a8cd74e3cb2f96b9243b5b2ef, State: Initialized, Role: FOLLOWER
I20260926 20:11:00.844605 31828 consensus_queue.cc:260] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.845382 31851 ts_tablet_manager.cc:1918] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:11:00.845503 31851 log.cc:1196] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/e485cce8e08e4c2e874c439b03f5059e
I20260926 20:11:00.845659 31828 ts_tablet_manager.cc:1428] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:11:00.845947 31851 ts_tablet_manager.cc:1939] T e485cce8e08e4c2e874c439b03f5059e P 8a6e1d9ca84c4e339b168d7042d5d08d: Deleting consensus metadata
I20260926 20:11:00.847376 31302 catalog_manager.cc:4841] TS 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463): tablet e485cce8e08e4c2e874c439b03f5059e (table test-table [id=52a329d26adb49c9a9a2223786ab5aa3]) successfully deleted
I20260926 20:11:00.848276 31830 tablet_bootstrap.cc:492] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d: No bootstrap required, opened a new log
I20260926 20:11:00.848387 31830 ts_tablet_manager.cc:1397] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260926 20:11:00.848834 31830 raft_consensus.cc:348] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.848938 31830 raft_consensus.cc:374] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:00.848989 31830 raft_consensus.cc:729] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a6e1d9ca84c4e339b168d7042d5d08d, State: Initialized, Role: FOLLOWER
I20260926 20:11:00.849126 31830 consensus_queue.cc:260] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:00.849428 31830 ts_tablet_manager.cc:1428] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260926 20:11:00.850708 31660 tablet.cc:2369] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:11:00.862516 31520 tablet.cc:2369] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:11:01.089828 31800 tablet.cc:2369] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:11:01.127070 31835 raft_consensus.cc:482] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:11:01.127239 31835 raft_consensus.cc:504] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.128448 31835 leader_election.cc:288] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795), dd62bd9a8cd74e3cb2f96b9243b5b2ef (127.23.50.130:33535)
I20260926 20:11:01.133116 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.128715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:44795 (local address 127.23.50.131:53199)
0926 20:11:01.129097 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:01.129114 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:01.129269 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:01.129731 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:01.129734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:01.129756 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:01.129930 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.129941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.130928 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.130930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.131804 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.131814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.132566 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.132572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.132721 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.132759 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:11:01.132841 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:11:01.132937 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260926 20:11:01.133105 31822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.128929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:53199 (local address 127.23.50.129:44795)
0926 20:11:01.129051 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.129056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.129209 (+ 153us) server_negotiation.cc:407] Connection header received
0926 20:11:01.129450 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.129457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.129512 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.129565 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:01.130040 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.130796 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.131939 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.132415 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.132519 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.132562 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.132879 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.132978 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:11:01.133172 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.128715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:33535 (local address 127.23.50.131:53367)
0926 20:11:01.129210 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:01.129223 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:01.129311 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:01.129624 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:01.129627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:01.129657 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:01.129805 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.129812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.130948 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.130951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.131845 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.131855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.132665 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.132668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.132804 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.132848 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 20:11:01.132964 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 20:11:01.133034 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":138,"thread_start_us":44,"threads_started":1}
I20260926 20:11:01.133208 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.128860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:53367 (local address 127.23.50.130:33535)
0926 20:11:01.128997 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.129003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.129300 (+ 297us) server_negotiation.cc:407] Connection header received
0926 20:11:01.129434 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.129438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.129513 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.129607 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:01.129983 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.130788 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.132014 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.132533 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.132663 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.132722 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.132975 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.133052 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:11:01.133888 31613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca966abb4d8414ba94db205b347a5aa" candidate_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" is_pre_election: true
I20260926 20:11:01.134121 31461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca966abb4d8414ba94db205b347a5aa" candidate_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" is_pre_election: true
I20260926 20:11:01.134217 31613 raft_consensus.cc:2454] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a6e1d9ca84c4e339b168d7042d5d08d in term 0.
I20260926 20:11:01.134284 31461 raft_consensus.cc:2454] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a6e1d9ca84c4e339b168d7042d5d08d in term 0.
I20260926 20:11:01.134749 31685 leader_election.cc:302] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8a6e1d9ca84c4e339b168d7042d5d08d, 9d299ab7cc8e4d33868dea282d0c2e4c, dd62bd9a8cd74e3cb2f96b9243b5b2ef; no voters:
I20260926 20:11:01.135037 31835 raft_consensus.cc:2781] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:11:01.135097 31835 raft_consensus.cc:482] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:11:01.135135 31835 raft_consensus.cc:3037] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:01.136132 31835 raft_consensus.cc:504] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.136516 31835 leader_election.cc:288] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [CANDIDATE]: Term 1 election: Requested vote from peers 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795), dd62bd9a8cd74e3cb2f96b9243b5b2ef (127.23.50.130:33535)
I20260926 20:11:01.136876 31461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca966abb4d8414ba94db205b347a5aa" candidate_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c"
I20260926 20:11:01.136937 31613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca966abb4d8414ba94db205b347a5aa" candidate_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef"
I20260926 20:11:01.136997 31461 raft_consensus.cc:3037] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:01.137105 31613 raft_consensus.cc:3037] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:01.138019 31461 raft_consensus.cc:2454] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a6e1d9ca84c4e339b168d7042d5d08d in term 1.
I20260926 20:11:01.138281 31613 raft_consensus.cc:2454] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a6e1d9ca84c4e339b168d7042d5d08d in term 1.
I20260926 20:11:01.138319 31685 leader_election.cc:302] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a6e1d9ca84c4e339b168d7042d5d08d, 9d299ab7cc8e4d33868dea282d0c2e4c; no voters:
I20260926 20:11:01.138484 31835 raft_consensus.cc:2781] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:11:01.138674 31835 raft_consensus.cc:686] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 LEADER]: Becoming Leader. State: Replica: 8a6e1d9ca84c4e339b168d7042d5d08d, State: Running, Role: LEADER
I20260926 20:11:01.138931 31835 consensus_queue.cc:237] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.141341 31316 catalog_manager.cc:5495] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131). New cstate: current_term: 1 leader_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } health_report { overall_health: HEALTHY } } }
Sep 26 20:11:01 dist-test-slave-7034 krb5kdc[31271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453459, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.190@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
I20260926 20:11:01.228672 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.219691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:39359 (local address 127.0.0.1:37812)
0926 20:11:01.220057 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:01.220071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:01.220220 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:01.220487 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:01.220489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:01.220903 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:01.220995 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.221000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.221804 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.221807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.222390 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.222397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.222877 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.222879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.222954 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.223571 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:01.223583 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:01.225080 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:01.227049 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:01.227057 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:01.227062 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:01.227317 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:01.227640 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:01.227642 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:01.227644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:01.227732 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:01.228133 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:01.228139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:01.228285 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 20:11:01.228400 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:11:01.228524 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":135,"threads_started":1}
I20260926 20:11:01.228822 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.219742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.23.50.190:39359)
0926 20:11:01.220072 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.220079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.220149 (+ 70us) server_negotiation.cc:407] Connection header received
0926 20:11:01.220309 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.220311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.220355 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.220434 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:01.221113 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.221683 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.222532 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.222757 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.222841 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.225265 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:01.225268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:01.225270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:01.225291 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:01.226933 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:01.227429 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:01.227432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:01.227433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:01.227527 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:01.227820 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:01.227822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:01.227823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:01.228001 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:01.228080 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.228489 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.228614 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":143,"threads_started":1}
I20260926 20:11:01.233227 31316 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:37812:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 20:11:01.234045 31316 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:11:01.239257 31593 tablet_service.cc:1467] Processing CreateTablet for tablet d72dcbc644b2431ba62ebd0dc42f4529 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=af53c148dc4c43ca9ff865af3f2e3745]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:11:01.239295 31453 tablet_service.cc:1467] Processing CreateTablet for tablet d72dcbc644b2431ba62ebd0dc42f4529 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=af53c148dc4c43ca9ff865af3f2e3745]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:11:01.239504 31733 tablet_service.cc:1467] Processing CreateTablet for tablet d72dcbc644b2431ba62ebd0dc42f4529 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=af53c148dc4c43ca9ff865af3f2e3745]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 20:11:01.239681 31593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72dcbc644b2431ba62ebd0dc42f4529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:01.239681 31453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72dcbc644b2431ba62ebd0dc42f4529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:01.239902 31733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72dcbc644b2431ba62ebd0dc42f4529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:01.242458 31828 tablet_bootstrap.cc:492] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Bootstrap starting.
I20260926 20:11:01.242589 31830 tablet_bootstrap.cc:492] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d: Bootstrap starting.
I20260926 20:11:01.242589 31827 tablet_bootstrap.cc:492] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: Bootstrap starting.
I20260926 20:11:01.243194 31828 tablet_bootstrap.cc:654] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:01.243553 31827 tablet_bootstrap.cc:654] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:01.243553 31830 tablet_bootstrap.cc:654] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:01.244326 31828 tablet_bootstrap.cc:492] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: No bootstrap required, opened a new log
I20260926 20:11:01.244424 31828 ts_tablet_manager.cc:1397] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:11:01.244922 31827 tablet_bootstrap.cc:492] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: No bootstrap required, opened a new log
I20260926 20:11:01.244927 31828 raft_consensus.cc:348] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.245040 31827 ts_tablet_manager.cc:1397] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:11:01.245050 31828 raft_consensus.cc:374] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:01.245087 31830 tablet_bootstrap.cc:492] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d: No bootstrap required, opened a new log
I20260926 20:11:01.245087 31828 raft_consensus.cc:729] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd62bd9a8cd74e3cb2f96b9243b5b2ef, State: Initialized, Role: FOLLOWER
I20260926 20:11:01.245173 31830 ts_tablet_manager.cc:1397] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:11:01.245226 31828 consensus_queue.cc:260] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.245509 31828 ts_tablet_manager.cc:1428] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:11:01.245524 31827 raft_consensus.cc:348] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.245636 31830 raft_consensus.cc:348] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.245715 31827 raft_consensus.cc:374] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:01.245750 31830 raft_consensus.cc:374] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:01.245757 31827 raft_consensus.cc:729] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d299ab7cc8e4d33868dea282d0c2e4c, State: Initialized, Role: FOLLOWER
I20260926 20:11:01.245800 31830 raft_consensus.cc:729] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a6e1d9ca84c4e339b168d7042d5d08d, State: Initialized, Role: FOLLOWER
I20260926 20:11:01.245759 31837 tablet_replica.cc:439] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } } }
I20260926 20:11:01.245913 31837 tablet_replica.cc:442] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:11:01.245939 31827 consensus_queue.cc:260] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.245955 31830 consensus_queue.cc:260] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.246208 31830 ts_tablet_manager.cc:1428] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:11:01.246250 31833 tablet_replica.cc:439] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } } }
I20260926 20:11:01.246380 31833 tablet_replica.cc:442] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:11:01.246398 31835 tablet_replica.cc:439] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } } }
I20260926 20:11:01.246517 31835 tablet_replica.cc:442] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:11:01.247009 31827 ts_tablet_manager.cc:1428] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 20:11:01.454933 31833 raft_consensus.cc:482] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:11:01.455123 31833 raft_consensus.cc:504] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.456264 31833 leader_election.cc:288] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd62bd9a8cd74e3cb2f96b9243b5b2ef (127.23.50.130:33535), 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463)
I20260926 20:11:01.460704 31869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.456497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:33535 (local address 127.23.50.129:45789)
0926 20:11:01.456760 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:01.456777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:01.456898 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:01.457450 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:01.457453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:01.457474 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:01.457639 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.457647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.458524 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.458528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.459515 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.459526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.460235 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.460240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.460347 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.460372 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:11:01.460453 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:11:01.460516 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
I20260926 20:11:01.460783 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.456601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:45789 (local address 127.23.50.130:33535)
0926 20:11:01.456744 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.456748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.456877 (+ 129us) server_negotiation.cc:407] Connection header received
0926 20:11:01.457274 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.457277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.457330 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.457415 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:01.457765 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.458394 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.459670 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.460094 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.460194 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.460234 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.460518 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.460611 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:11:01.460983 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.457053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:37451 (local address 127.23.50.131:37463)
0926 20:11:01.457318 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.457322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.457362 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:11:01.457444 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.457447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.457501 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.457592 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:01.457954 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.458581 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.459711 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.460187 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.460315 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.460371 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.460721 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.460822 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":88,"threads_started":1}
I20260926 20:11:01.461194 31613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72dcbc644b2431ba62ebd0dc42f4529" candidate_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" is_pre_election: true
I20260926 20:11:01.461339 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.456909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:37463 (local address 127.23.50.129:37451)
0926 20:11:01.457186 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:01.457201 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:01.457300 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:01.457616 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:01.457619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:01.457635 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:01.457779 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.457786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.458760 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.458763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.459543 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.459553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.460325 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.460329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.460569 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.460592 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:11:01.460681 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:11:01.460749 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260926 20:11:01.461385 31613 raft_consensus.cc:2454] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d299ab7cc8e4d33868dea282d0c2e4c in term 0.
I20260926 20:11:01.461792 31406 leader_election.cc:302] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d299ab7cc8e4d33868dea282d0c2e4c, dd62bd9a8cd74e3cb2f96b9243b5b2ef; no voters:
I20260926 20:11:01.461947 31753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72dcbc644b2431ba62ebd0dc42f4529" candidate_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" is_pre_election: true
I20260926 20:11:01.462126 31753 raft_consensus.cc:2454] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d299ab7cc8e4d33868dea282d0c2e4c in term 0.
I20260926 20:11:01.462137 31833 raft_consensus.cc:2781] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:11:01.462218 31833 raft_consensus.cc:482] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:11:01.462275 31833 raft_consensus.cc:3037] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:01.463327 31833 raft_consensus.cc:504] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.463670 31833 leader_election.cc:288] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [CANDIDATE]: Term 1 election: Requested vote from peers dd62bd9a8cd74e3cb2f96b9243b5b2ef (127.23.50.130:33535), 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463)
I20260926 20:11:01.464118 31613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72dcbc644b2431ba62ebd0dc42f4529" candidate_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef"
I20260926 20:11:01.464268 31613 raft_consensus.cc:3037] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:01.464232 31753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72dcbc644b2431ba62ebd0dc42f4529" candidate_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d"
I20260926 20:11:01.464409 31753 raft_consensus.cc:3037] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:01.465377 31613 raft_consensus.cc:2454] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d299ab7cc8e4d33868dea282d0c2e4c in term 1.
I20260926 20:11:01.465533 31753 raft_consensus.cc:2454] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d299ab7cc8e4d33868dea282d0c2e4c in term 1.
I20260926 20:11:01.465698 31406 leader_election.cc:302] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d299ab7cc8e4d33868dea282d0c2e4c, dd62bd9a8cd74e3cb2f96b9243b5b2ef; no voters:
I20260926 20:11:01.465919 31833 raft_consensus.cc:2781] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:11:01.466110 31833 raft_consensus.cc:686] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 LEADER]: Becoming Leader. State: Replica: 9d299ab7cc8e4d33868dea282d0c2e4c, State: Running, Role: LEADER
I20260926 20:11:01.466362 31833 consensus_queue.cc:237] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.467553 31872 tablet_replica.cc:439] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: TxnStatusTablet state changed. Reason: New leader 9d299ab7cc8e4d33868dea282d0c2e4c. Latest consensus state: current_term: 1 leader_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } } }
I20260926 20:11:01.467685 31872 tablet_replica.cc:442] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:11:01.467971 31874 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 20:11:01.468140 31874 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 20:11:01.468885 31314 catalog_manager.cc:5495] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129). New cstate: current_term: 1 leader_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } health_report { overall_health: UNKNOWN } } }
I20260926 20:11:01.488548 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.484528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:44795 (local address 127.0.0.1:35616)
0926 20:11:01.484781 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:01.484795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:01.484932 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:01.485334 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:01.485337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:01.485345 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:11:01.485460 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.485465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.486272 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.486279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.486916 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.486925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.487385 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.487388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.487491 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.487542 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:11:01.488250 (+ 708us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:11:01.488266 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 20:11:01.488327 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:11:01.488416 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 20:11:01.488548 31822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.484772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35616 (local address 127.23.50.129:44795)
0926 20:11:01.484919 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.484924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.484939 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:11:01.485144 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.485147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.485205 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.485310 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:11:01.485573 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.486154 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.487046 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.487273 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.487384 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.487704 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:11:01.488150 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:11:01.488279 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.488346 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.488403 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:11:01.491894 31753 raft_consensus.cc:1261] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Refusing update from remote peer 9d299ab7cc8e4d33868dea282d0c2e4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:11:01.491999 31613 raft_consensus.cc:1261] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Refusing update from remote peer 9d299ab7cc8e4d33868dea282d0c2e4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:11:01.492468 31833 consensus_queue.cc:1035] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:11:01.492903 31872 consensus_queue.cc:1035] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:11:01.494664 31835 tablet_replica.cc:439] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d: TxnStatusTablet state changed. Reason: New leader 9d299ab7cc8e4d33868dea282d0c2e4c. Latest consensus state: current_term: 1 leader_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } } }
I20260926 20:11:01.494783 31835 tablet_replica.cc:442] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:11:01.494724 31837 tablet_replica.cc:439] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: TxnStatusTablet state changed. Reason: New leader 9d299ab7cc8e4d33868dea282d0c2e4c. Latest consensus state: current_term: 1 leader_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } } }
I20260926 20:11:01.494845 31837 tablet_replica.cc:442] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:11:01.497462 31833 tablet_replica.cc:439] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } } }
I20260926 20:11:01.497587 31833 tablet_replica.cc:442] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:11:01.498472 31837 tablet_replica.cc:439] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } } }
I20260926 20:11:01.498582 31837 tablet_replica.cc:442] T d72dcbc644b2431ba62ebd0dc42f4529 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:11:01.499279 31835 tablet_replica.cc:439] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } } }
I20260926 20:11:01.499387 31835 tablet_replica.cc:442] T d72dcbc644b2431ba62ebd0dc42f4529 P 8a6e1d9ca84c4e339b168d7042d5d08d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 20:11:01.501506 31872 tablet_replica.cc:439] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } } }
I20260926 20:11:01.501660 31872 tablet_replica.cc:442] T d72dcbc644b2431ba62ebd0dc42f4529 P 9d299ab7cc8e4d33868dea282d0c2e4c: This TxnStatusTablet replica's current role is: LEADER
I20260926 20:11:01.501823 31878 mvcc.cc:205] Tried to move back new op lower bound from 7333697378265042944 to 7333697378168512512. Current Snapshot: MvccSnapshot[applied={T|T < 7333697378265042944}]
I20260926 20:11:01.507339 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.502915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.23.50.131:37463)
0926 20:11:01.503034 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.503039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.503273 (+ 234us) server_negotiation.cc:407] Connection header received
0926 20:11:01.503407 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.503409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.503459 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.503545 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:11:01.503859 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.504457 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.505743 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.506031 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.506111 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.506408 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:11:01.506948 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:11:01.507036 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.507141 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.507189 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:11:01.518622 31822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.514774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:49015 (local address 127.23.50.129:44795)
0926 20:11:01.514913 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.514918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.515095 (+ 177us) server_negotiation.cc:407] Connection header received
0926 20:11:01.515265 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.515268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.515349 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.515422 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:11:01.515708 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.516184 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.517429 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.517675 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.517771 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.518199 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:11:01.518271 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:11:01.518336 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.518459 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.518512 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:11:01.518707 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.514672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:44795 (local address 127.23.50.131:49015)
0926 20:11:01.515007 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:01.515022 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:01.515129 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:01.515455 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:01.515458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:01.515467 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:11:01.515585 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.515591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.516287 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.516291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.517295 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.517305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.517812 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.517816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.517941 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.517998 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:11:01.518371 (+ 373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:11:01.518377 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 20:11:01.518453 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:11:01.518544 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":109,"threads_started":1}
I20260926 20:11:01.526535 31613 raft_consensus.cc:1261] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Refusing update from remote peer 8a6e1d9ca84c4e339b168d7042d5d08d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:11:01.526631 31461 raft_consensus.cc:1261] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Refusing update from remote peer 8a6e1d9ca84c4e339b168d7042d5d08d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:11:01.527194 31857 consensus_queue.cc:1035] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:11:01.527391 31835 consensus_queue.cc:1035] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:11:01.551007 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.546325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:37463 (local address 127.23.50.129:56783)
0926 20:11:01.546715 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:01.546732 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:01.547012 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:01.547327 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:01.547329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:01.547358 (+ 29us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:11:01.547514 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.547521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.548590 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.548594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.549488 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.549498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.550312 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.550315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.550388 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.550436 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:11:01.550743 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:11:01.550748 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:11:01.550796 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 20:11:01.550861 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
I20260926 20:11:01.550999 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.546411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:56783 (local address 127.23.50.131:37463)
0926 20:11:01.546520 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.546525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.546803 (+ 278us) server_negotiation.cc:407] Connection header received
0926 20:11:01.547183 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.547186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.547243 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.547340 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:11:01.547624 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.548441 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.549607 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.550194 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.550298 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.550568 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:11:01.550649 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:11:01.550720 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.550813 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.550869 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 20:11:01.582290 31314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:11:01.582501 31314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:11:01.585160 31314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf: Sending DeleteTablet for 3 replicas of tablet eca966abb4d8414ba94db205b347a5aa
I20260926 20:11:01.586252 31733 tablet_service.cc:1514] Processing DeleteTablet for tablet eca966abb4d8414ba94db205b347a5aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:01 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:60392
I20260926 20:11:01.586292 31593 tablet_service.cc:1514] Processing DeleteTablet for tablet eca966abb4d8414ba94db205b347a5aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:01 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:33026
I20260926 20:11:01.586817 31453 tablet_service.cc:1514] Processing DeleteTablet for tablet eca966abb4d8414ba94db205b347a5aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:01 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:35614
I20260926 20:11:01.587222 31899 tablet_replica.cc:330] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef: stopping tablet replica
I20260926 20:11:01.587510 31899 raft_consensus.cc:2229] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:11:01.587666 31899 raft_consensus.cc:2258] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:01.588479 31899 ts_tablet_manager.cc:1905] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:01.591668 31899 ts_tablet_manager.cc:1918] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:11:01.591766 31899 log.cc:1196] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/eca966abb4d8414ba94db205b347a5aa
I20260926 20:11:01.592044 31899 ts_tablet_manager.cc:1939] T eca966abb4d8414ba94db205b347a5aa P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Deleting consensus metadata
I20260926 20:11:01.592898 31900 tablet_replica.cc:330] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c: stopping tablet replica
I20260926 20:11:01.592864 31302 catalog_manager.cc:4841] TS dd62bd9a8cd74e3cb2f96b9243b5b2ef (127.23.50.130:33535): tablet eca966abb4d8414ba94db205b347a5aa (table test-table [id=9b812acf614f4b4bad799623bd671c6e]) successfully deleted
I20260926 20:11:01.593147 31900 raft_consensus.cc:2229] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:11:01.593266 31898 tablet_replica.cc:330] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d: stopping tablet replica
I20260926 20:11:01.593371 31900 raft_consensus.cc:2258] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:01.593408 31898 raft_consensus.cc:2229] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:11:01.593881 31898 raft_consensus.cc:2258] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:01.594302 31900 ts_tablet_manager.cc:1905] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:01.597566 31900 ts_tablet_manager.cc:1918] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:11:01.597728 31900 log.cc:1196] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/wal/wals/eca966abb4d8414ba94db205b347a5aa
I20260926 20:11:01.598116 31900 ts_tablet_manager.cc:1939] T eca966abb4d8414ba94db205b347a5aa P 9d299ab7cc8e4d33868dea282d0c2e4c: Deleting consensus metadata
I20260926 20:11:01.598987 31301 catalog_manager.cc:4841] TS 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795): tablet eca966abb4d8414ba94db205b347a5aa (table test-table [id=9b812acf614f4b4bad799623bd671c6e]) successfully deleted
I20260926 20:11:01.605811 31898 ts_tablet_manager.cc:1905] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:01.609349 31898 ts_tablet_manager.cc:1918] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:11:01.609476 31898 log.cc:1196] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/eca966abb4d8414ba94db205b347a5aa
I20260926 20:11:01.609766 31898 ts_tablet_manager.cc:1939] T eca966abb4d8414ba94db205b347a5aa P 8a6e1d9ca84c4e339b168d7042d5d08d: Deleting consensus metadata
I20260926 20:11:01.610702 31302 catalog_manager.cc:4841] TS 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463): tablet eca966abb4d8414ba94db205b347a5aa (table test-table [id=9b812acf614f4b4bad799623bd671c6e]) successfully deleted
Sep 26 20:11:01 dist-test-slave-7034 krb5kdc[31271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 20:11:01 dist-test-slave-7034 krb5kdc[31271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:01 dist-test-slave-7034 krb5kdc[31271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:11:01.634996 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:01.626984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.23.50.190:39359)
0926 20:11:01.627127 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.627133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.627789 (+ 656us) server_negotiation.cc:407] Connection header received
0926 20:11:01.628009 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.628013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.628064 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.628149 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:01.629251 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.630042 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.631001 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.631268 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.631395 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.634884 (+ 3489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37814: BlockingRecv error: recv got EOF from 127.0.0.1:37814 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 20:11:01 dist-test-slave-7034 krb5kdc[31271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM
I20260926 20:11:01.647748 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.637237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.23.50.190:39359)
0926 20:11:01.637368 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.637373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.637819 (+ 446us) server_negotiation.cc:407] Connection header received
0926 20:11:01.638013 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.638017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.638073 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.638199 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:01.638868 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.639440 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.640315 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.640557 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.640677 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.643321 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:01.643325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:01.643327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:01.643354 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:01.644954 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:01.645441 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:01.645445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:01.645448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:01.645549 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:01.645988 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:01.645991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:01.645994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:01.646262 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:01.646436 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.647037 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.647358 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:11:01.652311 31314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:11:01.652837 31314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:11:01.658438 31453 tablet_service.cc:1467] Processing CreateTablet for tablet e7bd85c599b14bc7b9f712265bd27776 (DEFAULT_TABLE table=test-table [id=24792ce0c3af4f10aeff166252d7c669]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:01.658835 31453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7bd85c599b14bc7b9f712265bd27776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:01.659000 31593 tablet_service.cc:1467] Processing CreateTablet for tablet e7bd85c599b14bc7b9f712265bd27776 (DEFAULT_TABLE table=test-table [id=24792ce0c3af4f10aeff166252d7c669]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:01.659296 31593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7bd85c599b14bc7b9f712265bd27776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:01.661804 31828 tablet_bootstrap.cc:492] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Bootstrap starting.
I20260926 20:11:01.662055 31733 tablet_service.cc:1467] Processing CreateTablet for tablet e7bd85c599b14bc7b9f712265bd27776 (DEFAULT_TABLE table=test-table [id=24792ce0c3af4f10aeff166252d7c669]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:01.662349 31733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7bd85c599b14bc7b9f712265bd27776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:01.662786 31828 tablet_bootstrap.cc:654] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:01.663698 31827 tablet_bootstrap.cc:492] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c: Bootstrap starting.
I20260926 20:11:01.664646 31827 tablet_bootstrap.cc:654] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:01.665410 31830 tablet_bootstrap.cc:492] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d: Bootstrap starting.
I20260926 20:11:01.666443 31830 tablet_bootstrap.cc:654] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:01.667541 31828 tablet_bootstrap.cc:492] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: No bootstrap required, opened a new log
I20260926 20:11:01.667658 31828 ts_tablet_manager.cc:1397] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 20:11:01.667893 31827 tablet_bootstrap.cc:492] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c: No bootstrap required, opened a new log
I20260926 20:11:01.667975 31827 ts_tablet_manager.cc:1397] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:11:01.668264 31830 tablet_bootstrap.cc:492] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d: No bootstrap required, opened a new log
I20260926 20:11:01.668202 31828 raft_consensus.cc:348] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.668345 31830 ts_tablet_manager.cc:1397] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 20:11:01.668356 31828 raft_consensus.cc:374] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:01.668397 31828 raft_consensus.cc:729] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd62bd9a8cd74e3cb2f96b9243b5b2ef, State: Initialized, Role: FOLLOWER
I20260926 20:11:01.668390 31827 raft_consensus.cc:348] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.668504 31827 raft_consensus.cc:374] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:01.668534 31827 raft_consensus.cc:729] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d299ab7cc8e4d33868dea282d0c2e4c, State: Initialized, Role: FOLLOWER
I20260926 20:11:01.668552 31828 consensus_queue.cc:260] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.668661 31827 consensus_queue.cc:260] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.668854 31828 ts_tablet_manager.cc:1428] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:11:01.668905 31827 ts_tablet_manager.cc:1428] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:11:01.668839 31830 raft_consensus.cc:348] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.668953 31830 raft_consensus.cc:374] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:01.668998 31830 raft_consensus.cc:729] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a6e1d9ca84c4e339b168d7042d5d08d, State: Initialized, Role: FOLLOWER
I20260926 20:11:01.669149 31830 consensus_queue.cc:260] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.669430 31830 ts_tablet_manager.cc:1428] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:11:01.693961 31837 raft_consensus.cc:482] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:11:01.694156 31837 raft_consensus.cc:504] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.694624 31837 leader_election.cc:288] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795), 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463)
I20260926 20:11:01.695057 31753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bd85c599b14bc7b9f712265bd27776" candidate_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" is_pre_election: true
I20260926 20:11:01.695060 31461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bd85c599b14bc7b9f712265bd27776" candidate_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" is_pre_election: true
I20260926 20:11:01.695214 31753 raft_consensus.cc:2454] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd62bd9a8cd74e3cb2f96b9243b5b2ef in term 0.
I20260926 20:11:01.695228 31461 raft_consensus.cc:2454] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd62bd9a8cd74e3cb2f96b9243b5b2ef in term 0.
I20260926 20:11:01.695577 31545 leader_election.cc:302] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8a6e1d9ca84c4e339b168d7042d5d08d, 9d299ab7cc8e4d33868dea282d0c2e4c, dd62bd9a8cd74e3cb2f96b9243b5b2ef; no voters:
I20260926 20:11:01.695815 31837 raft_consensus.cc:2781] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:11:01.695863 31837 raft_consensus.cc:482] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:11:01.695914 31837 raft_consensus.cc:3037] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:01.696915 31837 raft_consensus.cc:504] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.697284 31837 leader_election.cc:288] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [CANDIDATE]: Term 1 election: Requested vote from peers 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795), 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463)
I20260926 20:11:01.697605 31753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bd85c599b14bc7b9f712265bd27776" candidate_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d"
I20260926 20:11:01.697610 31461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bd85c599b14bc7b9f712265bd27776" candidate_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c"
I20260926 20:11:01.697728 31753 raft_consensus.cc:3037] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:01.697727 31461 raft_consensus.cc:3037] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:01.698839 31461 raft_consensus.cc:2454] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd62bd9a8cd74e3cb2f96b9243b5b2ef in term 1.
I20260926 20:11:01.698843 31753 raft_consensus.cc:2454] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd62bd9a8cd74e3cb2f96b9243b5b2ef in term 1.
I20260926 20:11:01.699112 31545 leader_election.cc:302] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d299ab7cc8e4d33868dea282d0c2e4c, dd62bd9a8cd74e3cb2f96b9243b5b2ef; no voters:
I20260926 20:11:01.699290 31837 raft_consensus.cc:2781] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:11:01.699496 31837 raft_consensus.cc:686] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 LEADER]: Becoming Leader. State: Replica: dd62bd9a8cd74e3cb2f96b9243b5b2ef, State: Running, Role: LEADER
I20260926 20:11:01.699668 31837 consensus_queue.cc:237] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.701536 31316 catalog_manager.cc:5495] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef reported cstate change: term changed from 0 to 1, leader changed from <none> to dd62bd9a8cd74e3cb2f96b9243b5b2ef (127.23.50.130). New cstate: current_term: 1 leader_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } health_report { overall_health: UNKNOWN } } }
I20260926 20:11:01.717159 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.713912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.23.50.130:33535)
0926 20:11:01.714037 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.714041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.714056 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:11:01.714204 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.714207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.714252 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.714336 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:11:01.714599 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.715110 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.716005 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.716305 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.716416 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.716690 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:11:01.716777 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:11:01.716855 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.716945 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.717003 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:11:01.719233 31461 raft_consensus.cc:1261] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Refusing update from remote peer dd62bd9a8cd74e3cb2f96b9243b5b2ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:11:01.719240 31753 raft_consensus.cc:1261] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Refusing update from remote peer dd62bd9a8cd74e3cb2f96b9243b5b2ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:11:01.719692 31837 consensus_queue.cc:1035] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:11:01.719853 31916 consensus_queue.cc:1035] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:11:01.731244 31312 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:11:01.731451 31312 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:11:01.733943 31312 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf: Sending DeleteTablet for 3 replicas of tablet e7bd85c599b14bc7b9f712265bd27776
I20260926 20:11:01.734623 31453 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bd85c599b14bc7b9f712265bd27776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:01 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:35614
I20260926 20:11:01.734825 31593 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bd85c599b14bc7b9f712265bd27776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:01 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:33026
I20260926 20:11:01.735016 31899 tablet_replica.cc:330] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: stopping tablet replica
I20260926 20:11:01.735021 31900 tablet_replica.cc:330] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c: stopping tablet replica
I20260926 20:11:01.735001 31733 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bd85c599b14bc7b9f712265bd27776 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:01 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:60392
I20260926 20:11:01.735152 31900 raft_consensus.cc:2229] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:11:01.735188 31898 tablet_replica.cc:330] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d: stopping tablet replica
I20260926 20:11:01.735172 31899 raft_consensus.cc:2229] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:11:01.735283 31898 raft_consensus.cc:2229] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:11:01.735311 31900 raft_consensus.cc:2258] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:01.735379 31899 raft_consensus.cc:2258] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:01.735409 31898 raft_consensus.cc:2258] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:01.735610 31312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:11:01.736095 31312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:11:01.736363 31900 ts_tablet_manager.cc:1905] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:01.736455 31898 ts_tablet_manager.cc:1905] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:01.739073 31900 ts_tablet_manager.cc:1918] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:11:01.739100 31898 ts_tablet_manager.cc:1918] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:11:01.739157 31900 log.cc:1196] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-0/wal/wals/e7bd85c599b14bc7b9f712265bd27776
I20260926 20:11:01.739180 31898 log.cc:1196] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/e7bd85c599b14bc7b9f712265bd27776
I20260926 20:11:01.739425 31900 ts_tablet_manager.cc:1939] T e7bd85c599b14bc7b9f712265bd27776 P 9d299ab7cc8e4d33868dea282d0c2e4c: Deleting consensus metadata
I20260926 20:11:01.739450 31898 ts_tablet_manager.cc:1939] T e7bd85c599b14bc7b9f712265bd27776 P 8a6e1d9ca84c4e339b168d7042d5d08d: Deleting consensus metadata
I20260926 20:11:01.740489 31302 catalog_manager.cc:4841] TS 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463): tablet e7bd85c599b14bc7b9f712265bd27776 (table test-table [id=24792ce0c3af4f10aeff166252d7c669]) successfully deleted
I20260926 20:11:01.741091 31899 ts_tablet_manager.cc:1905] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:01.741408 31301 catalog_manager.cc:4841] TS 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795): tablet e7bd85c599b14bc7b9f712265bd27776 (table test-table [id=24792ce0c3af4f10aeff166252d7c669]) successfully deleted
I20260926 20:11:01.742590 31453 tablet_service.cc:1467] Processing CreateTablet for tablet 80eed2d6a33d4c0f986a166ed6f28891 (DEFAULT_TABLE table=test-table [id=5cccdeb43b3b4f5db05ff1bd43f5e5a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:01.742882 31453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80eed2d6a33d4c0f986a166ed6f28891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:01.744006 31899 ts_tablet_manager.cc:1918] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:11:01.744105 31899 log.cc:1196] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/e7bd85c599b14bc7b9f712265bd27776
I20260926 20:11:01.744387 31899 ts_tablet_manager.cc:1939] T e7bd85c599b14bc7b9f712265bd27776 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Deleting consensus metadata
I20260926 20:11:01.745045 31827 tablet_bootstrap.cc:492] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c: Bootstrap starting.
I20260926 20:11:01.745236 31593 tablet_service.cc:1467] Processing CreateTablet for tablet 80eed2d6a33d4c0f986a166ed6f28891 (DEFAULT_TABLE table=test-table [id=5cccdeb43b3b4f5db05ff1bd43f5e5a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:01.745513 31593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80eed2d6a33d4c0f986a166ed6f28891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:01.745817 31827 tablet_bootstrap.cc:654] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:01.746940 31827 tablet_bootstrap.cc:492] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c: No bootstrap required, opened a new log
I20260926 20:11:01.747049 31827 ts_tablet_manager.cc:1397] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 20:11:01.747506 31827 raft_consensus.cc:348] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.747653 31827 raft_consensus.cc:374] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:01.747699 31827 raft_consensus.cc:729] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d299ab7cc8e4d33868dea282d0c2e4c, State: Initialized, Role: FOLLOWER
I20260926 20:11:01.747848 31827 consensus_queue.cc:260] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.748044 31828 tablet_bootstrap.cc:492] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Bootstrap starting.
I20260926 20:11:01.748167 31827 ts_tablet_manager.cc:1428] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 20:11:01.748984 31828 tablet_bootstrap.cc:654] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:01.750198 31733 tablet_service.cc:1467] Processing CreateTablet for tablet 80eed2d6a33d4c0f986a166ed6f28891 (DEFAULT_TABLE table=test-table [id=5cccdeb43b3b4f5db05ff1bd43f5e5a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:01.750500 31733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80eed2d6a33d4c0f986a166ed6f28891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:01.753121 31830 tablet_bootstrap.cc:492] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d: Bootstrap starting.
I20260926 20:11:01.753566 31828 tablet_bootstrap.cc:492] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: No bootstrap required, opened a new log
I20260926 20:11:01.753674 31828 ts_tablet_manager.cc:1397] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 20:11:01.754160 31828 raft_consensus.cc:348] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.754298 31828 raft_consensus.cc:374] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:01.754336 31828 raft_consensus.cc:729] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd62bd9a8cd74e3cb2f96b9243b5b2ef, State: Initialized, Role: FOLLOWER
I20260926 20:11:01.754472 31828 consensus_queue.cc:260] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.754539 31830 tablet_bootstrap.cc:654] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:01.754771 31828 ts_tablet_manager.cc:1428] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 20:11:01.756057 31302 catalog_manager.cc:4841] TS dd62bd9a8cd74e3cb2f96b9243b5b2ef (127.23.50.130:33535): tablet e7bd85c599b14bc7b9f712265bd27776 (table test-table [id=24792ce0c3af4f10aeff166252d7c669]) successfully deleted
I20260926 20:11:01.756764 31830 tablet_bootstrap.cc:492] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d: No bootstrap required, opened a new log
I20260926 20:11:01.756892 31830 ts_tablet_manager.cc:1397] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 20:11:01.757346 31830 raft_consensus.cc:348] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.757475 31830 raft_consensus.cc:374] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:01.757525 31830 raft_consensus.cc:729] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a6e1d9ca84c4e339b168d7042d5d08d, State: Initialized, Role: FOLLOWER
I20260926 20:11:01.757908 31830 consensus_queue.cc:260] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.758333 31830 ts_tablet_manager.cc:1428] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:11:01.767306 31916 raft_consensus.cc:482] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:11:01.767458 31916 raft_consensus.cc:504] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.767992 31916 leader_election.cc:288] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795), 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463)
I20260926 20:11:01.768391 31461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80eed2d6a33d4c0f986a166ed6f28891" candidate_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" is_pre_election: true
I20260926 20:11:01.768533 31753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80eed2d6a33d4c0f986a166ed6f28891" candidate_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" is_pre_election: true
I20260926 20:11:01.768584 31461 raft_consensus.cc:2454] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd62bd9a8cd74e3cb2f96b9243b5b2ef in term 0.
I20260926 20:11:01.768698 31753 raft_consensus.cc:2454] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd62bd9a8cd74e3cb2f96b9243b5b2ef in term 0.
I20260926 20:11:01.768997 31545 leader_election.cc:302] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8a6e1d9ca84c4e339b168d7042d5d08d, 9d299ab7cc8e4d33868dea282d0c2e4c, dd62bd9a8cd74e3cb2f96b9243b5b2ef; no voters:
I20260926 20:11:01.769201 31916 raft_consensus.cc:2781] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:11:01.769266 31916 raft_consensus.cc:482] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:11:01.769336 31916 raft_consensus.cc:3037] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:01.770565 31916 raft_consensus.cc:504] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.770994 31916 leader_election.cc:288] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [CANDIDATE]: Term 1 election: Requested vote from peers 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795), 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463)
I20260926 20:11:01.771375 31461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80eed2d6a33d4c0f986a166ed6f28891" candidate_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c"
I20260926 20:11:01.771514 31461 raft_consensus.cc:3037] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:01.771564 31753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80eed2d6a33d4c0f986a166ed6f28891" candidate_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d"
I20260926 20:11:01.771706 31753 raft_consensus.cc:3037] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:01.772544 31461 raft_consensus.cc:2454] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd62bd9a8cd74e3cb2f96b9243b5b2ef in term 1.
I20260926 20:11:01.772826 31753 raft_consensus.cc:2454] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd62bd9a8cd74e3cb2f96b9243b5b2ef in term 1.
I20260926 20:11:01.773103 31546 leader_election.cc:302] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a6e1d9ca84c4e339b168d7042d5d08d, dd62bd9a8cd74e3cb2f96b9243b5b2ef; no voters:
I20260926 20:11:01.773375 31916 raft_consensus.cc:2781] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:11:01.773550 31916 raft_consensus.cc:686] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 LEADER]: Becoming Leader. State: Replica: dd62bd9a8cd74e3cb2f96b9243b5b2ef, State: Running, Role: LEADER
I20260926 20:11:01.773700 31916 consensus_queue.cc:237] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } }
I20260926 20:11:01.775628 31312 catalog_manager.cc:5495] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef reported cstate change: term changed from 0 to 1, leader changed from <none> to dd62bd9a8cd74e3cb2f96b9243b5b2ef (127.23.50.130). New cstate: current_term: 1 leader_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd62bd9a8cd74e3cb2f96b9243b5b2ef" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 } health_report { overall_health: UNKNOWN } } }
I20260926 20:11:01.802088 31927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.798493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:44795 (local address 127.23.50.130:49615)
0926 20:11:01.798814 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:01.798829 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:01.798971 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:01.799255 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:01.799258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:01.799266 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:11:01.799377 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.799383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.800121 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.800127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.800777 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.800787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.801269 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.801273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.801363 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.801417 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:11:01.801731 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:11:01.801741 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 20:11:01.801837 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 20:11:01.801958 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":111,"threads_started":1}
I20260926 20:11:01.802075 31822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.798613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:49615 (local address 127.23.50.129:44795)
0926 20:11:01.798760 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.798765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.798916 (+ 151us) server_negotiation.cc:407] Connection header received
0926 20:11:01.799091 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.799094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.799158 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.799243 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:11:01.799497 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.800009 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.800910 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.801153 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.801264 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.801557 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:11:01.801644 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:11:01.801727 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.801874 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.801927 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 20:11:01.808413 31753 raft_consensus.cc:1261] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Refusing update from remote peer dd62bd9a8cd74e3cb2f96b9243b5b2ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:11:01.808691 31461 raft_consensus.cc:1261] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Refusing update from remote peer dd62bd9a8cd74e3cb2f96b9243b5b2ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:11:01.808832 31837 consensus_queue.cc:1035] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a6e1d9ca84c4e339b168d7042d5d08d" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:11:01.809141 31837 consensus_queue.cc:1035] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d299ab7cc8e4d33868dea282d0c2e4c" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:11:01.828562 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.824679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:52841 (local address 127.23.50.130:33535)
0926 20:11:01.824795 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:01.824799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:01.824897 (+ 98us) server_negotiation.cc:407] Connection header received
0926 20:11:01.824993 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:01.824995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:01.825037 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:01.825139 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:11:01.825439 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.826106 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.827112 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.827460 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.827560 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.828018 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:11:01.828106 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:11:01.828162 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:01.828361 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 20:11:01.828417 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:11:01.828562 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.824553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:33535 (local address 127.23.50.129:52841)
0926 20:11:01.824822 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:01.824836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:01.824919 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:01.825149 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:01.825152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:01.825161 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 20:11:01.825283 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.825293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.826302 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.826305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.826909 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:01.826918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:01.827608 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:01.827613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:01.827714 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:01.827753 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 20:11:01.828232 (+ 479us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 20:11:01.828239 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:11:01.828323 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:11:01.828423 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260926 20:11:01.849987 31312 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:11:01.850145 31312 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:11:01.852339 31312 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e103592e8212445c9f2cbb95fafbbccf: Sending DeleteTablet for 3 replicas of tablet 80eed2d6a33d4c0f986a166ed6f28891
I20260926 20:11:01.853025 31453 tablet_service.cc:1514] Processing DeleteTablet for tablet 80eed2d6a33d4c0f986a166ed6f28891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:01 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:35614
I20260926 20:11:01.853096 31593 tablet_service.cc:1514] Processing DeleteTablet for tablet 80eed2d6a33d4c0f986a166ed6f28891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:01 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:33026
I20260926 20:11:01.853137 31733 tablet_service.cc:1514] Processing DeleteTablet for tablet 80eed2d6a33d4c0f986a166ed6f28891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:01 UTC) from {username='oryx', principal='oryx/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:60392
I20260926 20:11:01.853307 31900 tablet_replica.cc:330] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c: stopping tablet replica
I20260926 20:11:01.853345 31899 tablet_replica.cc:330] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: stopping tablet replica
I20260926 20:11:01.853345 31898 tablet_replica.cc:330] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d: stopping tablet replica
I20260926 20:11:01.853439 31900 raft_consensus.cc:2229] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:11:01.853451 31899 raft_consensus.cc:2229] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:11:01.853451 31898 raft_consensus.cc:2229] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:11:01.853561 31900 raft_consensus.cc:2258] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:01.853569 31898 raft_consensus.cc:2258] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:01.853638 31899 raft_consensus.cc:2258] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:01.854422 31899 ts_tablet_manager.cc:1905] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:01.854422 31898 ts_tablet_manager.cc:1905] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:01.854683 31900 ts_tablet_manager.cc:1905] T 80eed2d6a33d4c0f986a166ed6f28891 P 9d299ab7cc8e4d33868dea282d0c2e4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:01.854756 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 31388
I20260926 20:11:01.859282 31898 ts_tablet_manager.cc:1918] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:11:01.859301 31899 ts_tablet_manager.cc:1918] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 20:11:01.859395 31898 log.cc:1196] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/80eed2d6a33d4c0f986a166ed6f28891
I20260926 20:11:01.859414 31899 log.cc:1196] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/80eed2d6a33d4c0f986a166ed6f28891
I20260926 20:11:01.859702 31899 ts_tablet_manager.cc:1939] T 80eed2d6a33d4c0f986a166ed6f28891 P dd62bd9a8cd74e3cb2f96b9243b5b2ef: Deleting consensus metadata
I20260926 20:11:01.859699 31898 ts_tablet_manager.cc:1939] T 80eed2d6a33d4c0f986a166ed6f28891 P 8a6e1d9ca84c4e339b168d7042d5d08d: Deleting consensus metadata
I20260926 20:11:01.860532 31302 catalog_manager.cc:4841] TS 8a6e1d9ca84c4e339b168d7042d5d08d (127.23.50.131:37463): tablet 80eed2d6a33d4c0f986a166ed6f28891 (table test-table [id=5cccdeb43b3b4f5db05ff1bd43f5e5a3]) successfully deleted
I20260926 20:11:01.860752 31302 catalog_manager.cc:4841] TS dd62bd9a8cd74e3cb2f96b9243b5b2ef (127.23.50.130:33535): tablet 80eed2d6a33d4c0f986a166ed6f28891 (table test-table [id=5cccdeb43b3b4f5db05ff1bd43f5e5a3]) successfully deleted
W20260926 20:11:01.864187 31301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:44795 (error 108)
I20260926 20:11:01.864365 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 31528
I20260926 20:11:01.866518 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:01.866035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:44795 (local address 127.0.0.1:35624)
0926 20:11:01.866342 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:01.866423 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:44795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":124,"threads_started":1}
W20260926 20:11:01.866797 31301 catalog_manager.cc:4584] TS 9d299ab7cc8e4d33868dea282d0c2e4c (127.23.50.129:44795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80eed2d6a33d4c0f986a166ed6f28891: Network error: Client connection negotiation failed: client connection to 127.23.50.129:44795: connect: Connection refused (error 111)
I20260926 20:11:01.871626 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 31668
I20260926 20:11:01.878136 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 31284
2026-09-26T20:11:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3464 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:11:01 dist-test-slave-7034 krb5kdc[31935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:11:01 dist-test-slave-7034 krb5kdc[31935](info): set up 2 sockets
Sep 26 20:11:01 dist-test-slave-7034 krb5kdc[31935](info): commencing operation
krb5kdc: starting...
W20260926 20:11:03.035168 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:11:03 dist-test-slave-7034 krb5kdc[31935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:11:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:11:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:11:03.165638 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:32811
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:32811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:03.284123 31948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:03.284446 31948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:03.284528 31948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:03.288578 31948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:11:03.288687 31948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:03.288728 31948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:03.288789 31948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:11:03.288813 31948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:11:03.294279 31948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:32811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:32811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:03.295389 31948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:03.296413 31948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:03.305188 31956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:03.305190 31954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:03.305259 31953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:03.305250 31948 server_base.cc:1034] running on GCE node
I20260926 20:11:03.306046 31948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:03.307121 31948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:03.308281 31948 hybrid_clock.cc:648] HybridClock initialized: now 1790453463308263 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T20:11:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1444 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:11:03 dist-test-slave-7034 krb5kdc[31961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:11:03 dist-test-slave-7034 krb5kdc[31961](info): set up 2 sockets
Sep 26 20:11:03 dist-test-slave-7034 krb5kdc[31961](info): commencing operation
krb5kdc: starting...
W20260926 20:11:04.371194 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:11:04 dist-test-slave-7034 krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:11:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:11:04.497465 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42825
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:04.609521 31974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:04.609889 31974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:04.610030 31974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:04.614084 31974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:11:04.614184 31974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:04.614223 31974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:04.614262 31974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:11:04.614306 31974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:11:04.619824 31974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:42825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:04.620929 31974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:04.621990 31974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:04.630705 31982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:04.630862 31974 server_base.cc:1034] running on GCE node
W20260926 20:11:04.630813 31980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:04.630813 31979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:04.631633 31974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:04.632750 31974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:04.633953 31974 hybrid_clock.cc:648] HybridClock initialized: now 1790453464633926 us; error 32 us; skew 500 ppm
Sep 26 20:11:04 dist-test-slave-7034 krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453464, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:04.636859 31974 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:11:04.637821 31974 webserver.cc:460] Webserver started at http://127.23.50.190:46587/ using document root <none> and password file <none>
I20260926 20:11:04.638447 31974 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:04.638537 31974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:04.638782 31974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:04.640841 31974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "a024e707e46143e3ac9aeb3fbc9f1017"
format_stamp: "Formatted at 2026-09-26 20:11:04 on dist-test-slave-7034"
server_key: "a946e3f567476c466e160b509485bad5"
server_key_iv: "f21cd90a56c9b92c4f3f6d348153edc9"
server_key_version: "encryptionkey@0"
I20260926 20:11:04.641417 31974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "a024e707e46143e3ac9aeb3fbc9f1017"
format_stamp: "Formatted at 2026-09-26 20:11:04 on dist-test-slave-7034"
server_key: "a946e3f567476c466e160b509485bad5"
server_key_iv: "f21cd90a56c9b92c4f3f6d348153edc9"
server_key_version: "encryptionkey@0"
I20260926 20:11:04.645329 31974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:11:04.647976 31989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:04.649346 31974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:11:04.649510 31974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "a024e707e46143e3ac9aeb3fbc9f1017"
format_stamp: "Formatted at 2026-09-26 20:11:04 on dist-test-slave-7034"
server_key: "a946e3f567476c466e160b509485bad5"
server_key_iv: "f21cd90a56c9b92c4f3f6d348153edc9"
server_key_version: "encryptionkey@0"
I20260926 20:11:04.649641 31974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:04.662587 31974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:04.666072 31974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:04.666288 31974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:04.673425 31974 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:42825
I20260926 20:11:04.673450 32041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:42825 every 8 connection(s)
I20260926 20:11:04.674465 31974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:11:04.677175 32042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:04.682914 32042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017: Bootstrap starting.
I20260926 20:11:04.684227 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 31974
I20260926 20:11:04.684343 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:11:04.684676 23754 external_mini_cluster.cc:1428] Setting key 836cc9df4d6d466c443c217abeaf90ff
I20260926 20:11:04.685626 32042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:04.686573 32042 log.cc:824] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017: Log is configured to *not* fsync() on all Append() calls
I20260926 20:11:04.688541 32042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017: No bootstrap required, opened a new log
I20260926 20:11:04.692049 32042 raft_consensus.cc:348] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42825 } }
I20260926 20:11:04.692312 32042 raft_consensus.cc:374] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:04.692401 32042 raft_consensus.cc:729] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a024e707e46143e3ac9aeb3fbc9f1017, State: Initialized, Role: FOLLOWER
I20260926 20:11:04.692955 32042 consensus_queue.cc:260] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42825 } }
I20260926 20:11:04.693142 32042 raft_consensus.cc:388] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:11:04.693217 32042 raft_consensus.cc:482] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:11:04.693349 32042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 20:11:04 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:11:04.694454 32042 raft_consensus.cc:504] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42825 } }
I20260926 20:11:04.694785 32042 leader_election.cc:302] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a024e707e46143e3ac9aeb3fbc9f1017; no voters:
I20260926 20:11:04.695125 32042 leader_election.cc:288] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:11:04.695314 32047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:11:04.695576 32047 raft_consensus.cc:686] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 1 LEADER]: Becoming Leader. State: Replica: a024e707e46143e3ac9aeb3fbc9f1017, State: Running, Role: LEADER
I20260926 20:11:04.695919 32047 consensus_queue.cc:237] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42825 } }
I20260926 20:11:04.696442 32042 sys_catalog.cc:564] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:11:04.697568 32048 sys_catalog.cc:455] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42825 } } }
I20260926 20:11:04.697692 32048 sys_catalog.cc:458] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [sys.catalog]: This master's current role is: LEADER
I20260926 20:11:04.697788 32049 sys_catalog.cc:455] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a024e707e46143e3ac9aeb3fbc9f1017. Latest consensus state: current_term: 1 leader_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42825 } } }
I20260926 20:11:04.697933 32049 sys_catalog.cc:458] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [sys.catalog]: This master's current role is: LEADER
I20260926 20:11:04.700649 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:04.686003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54806 (local address 127.23.50.190:42825)
0926 20:11:04.686477 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:04.686494 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:04.686533 (+ 39us) server_negotiation.cc:407] Connection header received
0926 20:11:04.687357 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:04.687376 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:04.687748 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:04.688149 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:04.689030 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:04.690190 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:04.691150 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:04.691692 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:04.691875 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:04.694882 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:04.694887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:04.694901 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:04.694952 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:04.697114 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:04.697649 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:04.697655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:04.697661 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:04.697837 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:04.698240 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:04.698244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:04.698248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:04.698767 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:04.698970 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:04.699584 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 20:11:04.699874 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":137,"threads_started":1}
W20260926 20:11:04.701726 32062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:11:04.701798 32062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:11:04.701884 32056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:11:04.702709 32056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:11:04.708841 32056 catalog_manager.cc:1347] Generated new cluster ID: bfb772e509ec4ad0a866874b7fd30f4b
I20260926 20:11:04.708911 32056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:11:04.752195 32056 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:11:04.753552 32056 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:11:04.760421 32056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017: Generated new TSK 0
I20260926 20:11:04.761245 32056 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:11:04.801139 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42825
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:04.938225 32068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:04.938520 32068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:04.938565 32068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:04.943166 32068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:04.943252 32068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:04.943320 32068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:11:04.948959 32068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:04.950222 32068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:04.951262 32068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:04.961402 32076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:04.961416 32073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:04.961565 32068 server_base.cc:1034] running on GCE node
W20260926 20:11:04.961426 32074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:04.962229 32068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:04.962970 32068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:04.964170 32068 hybrid_clock.cc:648] HybridClock initialized: now 1790453464964133 us; error 51 us; skew 500 ppm
Sep 26 20:11:04 dist-test-slave-7034 krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453464, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:04.967168 32068 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:11:04.968098 32068 webserver.cc:460] Webserver started at http://127.23.50.129:45241/ using document root <none> and password file <none>
I20260926 20:11:04.968685 32068 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:04.968777 32068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:04.969033 32068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:04.971238 32068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "4fd7de94815844e58b18d1961e838ea7"
format_stamp: "Formatted at 2026-09-26 20:11:04 on dist-test-slave-7034"
server_key: "eb2e9b1f0ec8b36fae92edd89a81548c"
server_key_iv: "0397c8d42a6ca66e272580a95ea437be"
server_key_version: "encryptionkey@0"
I20260926 20:11:04.971825 32068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "4fd7de94815844e58b18d1961e838ea7"
format_stamp: "Formatted at 2026-09-26 20:11:04 on dist-test-slave-7034"
server_key: "eb2e9b1f0ec8b36fae92edd89a81548c"
server_key_iv: "0397c8d42a6ca66e272580a95ea437be"
server_key_version: "encryptionkey@0"
I20260926 20:11:04.975857 32068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:11:04.978585 32083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:04.979920 32068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:11:04.980063 32068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "4fd7de94815844e58b18d1961e838ea7"
format_stamp: "Formatted at 2026-09-26 20:11:04 on dist-test-slave-7034"
server_key: "eb2e9b1f0ec8b36fae92edd89a81548c"
server_key_iv: "0397c8d42a6ca66e272580a95ea437be"
server_key_version: "encryptionkey@0"
I20260926 20:11:04.980230 32068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:05.007138 32068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:05.010731 32068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:05.010947 32068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:05.011564 32068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:05.012420 32068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:05.012488 32068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:05.012562 32068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:05.012605 32068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:05.022631 32068 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:35747
I20260926 20:11:05.022719 32196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:35747 every 8 connection(s)
I20260926 20:11:05.023630 32068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:11:05.031082 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 32068
I20260926 20:11:05.031232 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:11:05.031514 23754 external_mini_cluster.cc:1428] Setting key c104b13524e2994584b8c7f2b0ab7ea6
Sep 26 20:11:05 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453464, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:11:05.038738 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:05.025586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:38719 (local address 127.23.50.190:42825)
0926 20:11:05.025742 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:05.025746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:05.026736 (+ 990us) server_negotiation.cc:407] Connection header received
0926 20:11:05.028112 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:05.028115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:05.028173 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:05.028259 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:05.029466 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:05.030085 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:05.031241 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:05.031544 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:05.031673 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:05.034784 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:05.034788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:05.034790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:05.034816 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:05.036491 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:05.037079 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:05.037082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:05.037084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:05.037160 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:05.037528 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:05.037533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:05.037535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:05.037742 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:05.037856 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:05.038415 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 20:11:05.038555 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:11:05.039594 32199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:05.026071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:38719)
0926 20:11:05.026557 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:05.026604 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:05.027875 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:05.028442 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:05.028450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:05.028817 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:05.029311 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:05.029323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:05.030214 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:05.030218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:05.031095 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:05.031114 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:05.031821 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:05.031825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:05.031972 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:05.032584 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:05.032603 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:05.034536 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:05.036631 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:05.036638 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:05.036655 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:05.036953 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:05.037256 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:05.037259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:05.037262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:05.037416 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:05.037874 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:05.037882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:05.038162 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 20:11:05.038375 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 20:11:05.038739 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":101,"threads_started":1}
I20260926 20:11:05.040885 32197 heartbeater.cc:347] Connected to a master server at 127.23.50.190:42825
I20260926 20:11:05.041157 32197 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:11:05.041776 32197 heartbeater.cc:511] Master 127.23.50.190:42825 requested a full tablet report, sending...
I20260926 20:11:05.043443 32006 ts_manager.cc:194] Registered new tserver with Master: 4fd7de94815844e58b18d1961e838ea7 (127.23.50.129:35747)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
I20260926 20:11:05.044960 32006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:38719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:11:05.062243 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42825
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:05.176682 32202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:05.176977 32202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:05.177022 32202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:05.181320 32202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:05.181406 32202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:05.181468 32202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:11:05.187325 32202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:05.188544 32202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:05.189633 32202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:05.200089 32207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:05.200083 32208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:05.200225 32210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:05.200160 32202 server_base.cc:1034] running on GCE node
I20260926 20:11:05.200901 32202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:05.201598 32202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:05.202860 32202 hybrid_clock.cc:648] HybridClock initialized: now 1790453465202847 us; error 64 us; skew 500 ppm
Sep 26 20:11:05 dist-test-slave-7034 krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453465, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:05.206820 32202 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:11:05.208123 32202 webserver.cc:460] Webserver started at http://127.23.50.130:33185/ using document root <none> and password file <none>
I20260926 20:11:05.209220 32202 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:05.209339 32202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:05.209641 32202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:05.212388 32202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "45e24c5cad6342e4bedc34f6bf28eb19"
format_stamp: "Formatted at 2026-09-26 20:11:05 on dist-test-slave-7034"
server_key: "0761909a7aa71da572a995c1d3f756fb"
server_key_iv: "42e44ffcd09648d8e150f6641c61819a"
server_key_version: "encryptionkey@0"
I20260926 20:11:05.213194 32202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "45e24c5cad6342e4bedc34f6bf28eb19"
format_stamp: "Formatted at 2026-09-26 20:11:05 on dist-test-slave-7034"
server_key: "0761909a7aa71da572a995c1d3f756fb"
server_key_iv: "42e44ffcd09648d8e150f6641c61819a"
server_key_version: "encryptionkey@0"
I20260926 20:11:05.222819 32202 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.006s
I20260926 20:11:05.226823 32217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:05.228696 32202 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260926 20:11:05.228843 32202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "45e24c5cad6342e4bedc34f6bf28eb19"
format_stamp: "Formatted at 2026-09-26 20:11:05 on dist-test-slave-7034"
server_key: "0761909a7aa71da572a995c1d3f756fb"
server_key_iv: "42e44ffcd09648d8e150f6641c61819a"
server_key_version: "encryptionkey@0"
I20260926 20:11:05.229014 32202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:05.245924 32202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:05.250034 32202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:05.250245 32202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:05.250880 32202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:05.251781 32202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:05.251853 32202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:05.251921 32202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:05.251969 32202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:05.261374 32202 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:42533
I20260926 20:11:05.261392 32330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:42533 every 8 connection(s)
I20260926 20:11:05.262389 32202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:11:05.269906 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 32202
I20260926 20:11:05.270066 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:11:05.270381 23754 external_mini_cluster.cc:1428] Setting key 2d4bbab0508d378f5883bfebf9dd7cd1
Sep 26 20:11:05 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453465, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:11:05.276984 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:05.264289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:33633 (local address 127.23.50.190:42825)
0926 20:11:05.264453 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:05.264458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:05.265330 (+ 872us) server_negotiation.cc:407] Connection header received
0926 20:11:05.266718 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:05.266721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:05.266770 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:05.266848 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:05.268032 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:05.268570 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:05.269736 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:05.270039 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:05.270169 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:05.272984 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:05.272991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:05.272998 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:05.273033 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:05.274692 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:05.275348 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:05.275353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:05.275354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:05.275442 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:05.275816 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:05.275821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:05.275822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:05.276017 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:05.276129 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:05.276661 (+ 532us) server_negotiation.cc:299] Negotiation successful
0926 20:11:05.276808 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:11:05.277829 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:05.264713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:33633)
0926 20:11:05.265173 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:05.265219 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:05.266478 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:05.267020 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:05.267027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:05.267398 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:05.267904 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:05.267917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:05.268693 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:05.268695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:05.269602 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:05.269620 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:05.270214 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:05.270219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:05.270433 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:05.271142 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:05.271164 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:05.272810 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:05.274823 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:05.274828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:05.274843 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:05.275108 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:05.275542 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:05.275546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:05.275548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:05.275715 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:05.276126 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:05.276134 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:05.276412 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 20:11:05.276608 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 20:11:05.276957 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260926 20:11:05.279289 32331 heartbeater.cc:347] Connected to a master server at 127.23.50.190:42825
I20260926 20:11:05.279558 32331 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:11:05.280108 32331 heartbeater.cc:511] Master 127.23.50.190:42825 requested a full tablet report, sending...
I20260926 20:11:05.281330 32006 ts_manager.cc:194] Registered new tserver with Master: 45e24c5cad6342e4bedc34f6bf28eb19 (127.23.50.130:42533)
I20260926 20:11:05.282061 32006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:33633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:11:05.301203 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42825
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:05.416651 32336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:05.416935 32336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:05.417008 32336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:05.421272 32336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:05.421339 32336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:05.421432 32336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:11:05.427152 32336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:05.428416 32336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:05.429447 32336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:05.439446 32341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:05.439472 32344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:05.439617 32336 server_base.cc:1034] running on GCE node
W20260926 20:11:05.439810 32342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:05.440344 32336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:05.441002 32336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:05.442229 32336 hybrid_clock.cc:648] HybridClock initialized: now 1790453465442194 us; error 50 us; skew 500 ppm
Sep 26 20:11:05 dist-test-slave-7034 krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453465, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:05.445523 32336 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:11:05.446602 32336 webserver.cc:460] Webserver started at http://127.23.50.131:37425/ using document root <none> and password file <none>
I20260926 20:11:05.447197 32336 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:05.447258 32336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:05.447496 32336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:05.449728 32336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "ebc24cd550b545fca375d39d6426d0ba"
format_stamp: "Formatted at 2026-09-26 20:11:05 on dist-test-slave-7034"
server_key: "93508f201a497a805f330db578407a8a"
server_key_iv: "847564b6e99984cd3edf0d7541389d52"
server_key_version: "encryptionkey@0"
I20260926 20:11:05.450378 32336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "ebc24cd550b545fca375d39d6426d0ba"
format_stamp: "Formatted at 2026-09-26 20:11:05 on dist-test-slave-7034"
server_key: "93508f201a497a805f330db578407a8a"
server_key_iv: "847564b6e99984cd3edf0d7541389d52"
server_key_version: "encryptionkey@0"
I20260926 20:11:05.454381 32336 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:11:05.456915 32351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:05.458276 32336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:11:05.458417 32336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "ebc24cd550b545fca375d39d6426d0ba"
format_stamp: "Formatted at 2026-09-26 20:11:05 on dist-test-slave-7034"
server_key: "93508f201a497a805f330db578407a8a"
server_key_iv: "847564b6e99984cd3edf0d7541389d52"
server_key_version: "encryptionkey@0"
I20260926 20:11:05.458531 32336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:05.480264 32336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:05.483944 32336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:05.484156 32336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:05.484851 32336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:05.485689 32336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:05.485734 32336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:05.485803 32336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:05.485884 32336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:05.495036 32336 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:35795
I20260926 20:11:05.495059 32464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:35795 every 8 connection(s)
I20260926 20:11:05.496014 32336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:11:05.498309 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 32336
I20260926 20:11:05.498453 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:11:05.498754 23754 external_mini_cluster.cc:1428] Setting key b97aa50a306350aa7519279f526a50a0
Sep 26 20:11:05 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453465, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:11:05.509487 32045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:05.498089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:40221 (local address 127.23.50.190:42825)
0926 20:11:05.498237 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:05.498243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:05.499268 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 20:11:05.500599 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:05.500602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:05.500658 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:05.500749 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:05.501928 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:05.502512 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:05.503598 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:05.503812 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:05.503889 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:05.506256 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:05.506259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:05.506261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:05.506284 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:05.507582 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:05.508039 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:05.508041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:05.508042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:05.508128 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:05.508499 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:05.508501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:05.508503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:05.508688 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:05.508766 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:05.509260 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 20:11:05.509363 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:11:05.510377 32467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:05.498647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:40221)
0926 20:11:05.499116 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:05.499171 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:05.500405 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:05.500915 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:05.500922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:05.501268 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:05.501742 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:05.501755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:05.502620 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:05.502622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:05.503496 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:05.503504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:05.503936 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:05.503939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:05.504077 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:05.504603 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:05.504619 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:05.506104 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:05.507684 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:05.507688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:05.507699 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:05.507945 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:05.508260 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:05.508262 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:05.508264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:05.508412 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:05.508771 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:05.508776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:05.509007 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 20:11:05.509154 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 20:11:05.509419 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":138,"threads_started":1}
I20260926 20:11:05.511664 32465 heartbeater.cc:347] Connected to a master server at 127.23.50.190:42825
I20260926 20:11:05.511910 32465 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:05.512425 32465 heartbeater.cc:511] Master 127.23.50.190:42825 requested a full tablet report, sending...
I20260926 20:11:05.513458 32006 ts_manager.cc:194] Registered new tserver with Master: ebc24cd550b545fca375d39d6426d0ba (127.23.50.131:35795)
I20260926 20:11:05.514047 32006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:40221
I20260926 20:11:05.524966 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab.
Entry for principal oryx/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab.
WARNING: no policy specified for oryx/127.23.50.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab.
Entry for principal oryx/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab.
WARNING: no policy specified for oryx/127.23.50.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.50.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab.
Entry for principal oryx/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab.
I20260926 20:11:05.604143 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 32068
I20260926 20:11:05.609582 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 32202
I20260926 20:11:05.615069 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 32336
I20260926 20:11:05.620339 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 31974
I20260926 20:11:05.626780 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42825
--webserver_interface=127.23.50.190
--webserver_port=46587
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:05.748502 32474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:05.748847 32474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:05.748932 32474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:05.753104 32474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:11:05.753214 32474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:05.753255 32474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:05.753315 32474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:11:05.753353 32474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:11:05.759315 32474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:42825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=46587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:05.760510 32474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:05.761569 32474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:05.770483 32482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:05.770540 32480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:05.770519 32479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:05.770673 32474 server_base.cc:1034] running on GCE node
I20260926 20:11:05.771314 32474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:05.772488 32474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:05.773708 32474 hybrid_clock.cc:648] HybridClock initialized: now 1790453465773694 us; error 30 us; skew 500 ppm
Sep 26 20:11:05 dist-test-slave-7034 krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453465, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:05.776652 32474 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:11:05.777599 32474 webserver.cc:460] Webserver started at http://127.23.50.190:46587/ using document root <none> and password file <none>
I20260926 20:11:05.778264 32474 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:05.778385 32474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:05.782425 32474 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:11:05.784679 32489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:05.786058 32474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:11:05.786208 32474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "a024e707e46143e3ac9aeb3fbc9f1017"
format_stamp: "Formatted at 2026-09-26 20:11:04 on dist-test-slave-7034"
server_key: "a946e3f567476c466e160b509485bad5"
server_key_iv: "f21cd90a56c9b92c4f3f6d348153edc9"
server_key_version: "encryptionkey@0"
I20260926 20:11:05.786778 32474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:05.798406 32474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:05.801829 32474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:05.802098 32474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:05.809029 32474 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:42825
I20260926 20:11:05.809106 32541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:42825 every 8 connection(s)
I20260926 20:11:05.810151 32474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:11:05.813505 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 32474
I20260926 20:11:05.814028 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:35747
--local_ip_for_outbound_sockets=127.23.50.129
--tserver_master_addrs=127.23.50.190:42825
--webserver_port=45241
--webserver_interface=127.23.50.129
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.50.129
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:11:05.814821 32542 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:11:05.816808 32542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017: Bootstrap starting.
I20260926 20:11:05.834175 32542 log.cc:824] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017: Log is configured to *not* fsync() on all Append() calls
I20260926 20:11:05.840755 32542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:11:05.841152 32542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017: Bootstrap complete.
I20260926 20:11:05.845393 32542 raft_consensus.cc:348] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42825 } }
I20260926 20:11:05.845815 32542 raft_consensus.cc:729] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a024e707e46143e3ac9aeb3fbc9f1017, State: Initialized, Role: FOLLOWER
I20260926 20:11:05.846444 32542 consensus_queue.cc:260] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42825 } }
I20260926 20:11:05.846562 32542 raft_consensus.cc:388] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:11:05.846648 32542 raft_consensus.cc:482] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:11:05.846755 32542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:11:05.848786 32542 raft_consensus.cc:504] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42825 } }
I20260926 20:11:05.849126 32542 leader_election.cc:302] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a024e707e46143e3ac9aeb3fbc9f1017; no voters:
I20260926 20:11:05.849457 32542 leader_election.cc:288] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:11:05.849548 32546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:11:05.849732 32546 raft_consensus.cc:686] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [term 2 LEADER]: Becoming Leader. State: Replica: a024e707e46143e3ac9aeb3fbc9f1017, State: Running, Role: LEADER
I20260926 20:11:05.850046 32546 consensus_queue.cc:237] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42825 } }
I20260926 20:11:05.850730 32542 sys_catalog.cc:564] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:11:05.851042 32548 sys_catalog.cc:455] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a024e707e46143e3ac9aeb3fbc9f1017. Latest consensus state: current_term: 2 leader_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42825 } } }
I20260926 20:11:05.851174 32548 sys_catalog.cc:458] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [sys.catalog]: This master's current role is: LEADER
I20260926 20:11:05.851393 32547 sys_catalog.cc:455] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a024e707e46143e3ac9aeb3fbc9f1017" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42825 } } }
I20260926 20:11:05.851493 32547 sys_catalog.cc:458] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017 [sys.catalog]: This master's current role is: LEADER
I20260926 20:11:05.851549 32550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:11:05.855206 32550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:11:05.855850 32550 catalog_manager.cc:1259] Loaded cluster ID: bfb772e509ec4ad0a866874b7fd30f4b
I20260926 20:11:05.855914 32550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:11:05.858542 32550 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:11:05.859262 32550 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a024e707e46143e3ac9aeb3fbc9f1017: Loaded TSK: 0
I20260926 20:11:05.860505 32550 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:11:05.931722 32544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:05.932047 32544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:05.932124 32544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:05.936430 32544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:05.936537 32544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:05.936635 32544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:11:05.942297 32544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.129.keytab
--principal=oryx/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:35747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=45241
--tserver_master_addrs=127.23.50.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:05.943420 32544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:05.944514 32544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:05.955854 32568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:05.955876 32571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:05.955870 32569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:05.956209 32544 server_base.cc:1034] running on GCE node
I20260926 20:11:05.956664 32544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:05.957451 32544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:05.958664 32544 hybrid_clock.cc:648] HybridClock initialized: now 1790453465958644 us; error 38 us; skew 500 ppm
Sep 26 20:11:05 dist-test-slave-7034 krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453465, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:05.961629 32544 init.cc:377] Logged in from keytab as oryx/127.23.50.129@KRBTEST.COM (short username oryx)
I20260926 20:11:05.962692 32544 webserver.cc:460] Webserver started at http://127.23.50.129:45241/ using document root <none> and password file <none>
I20260926 20:11:05.963285 32544 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:05.963397 32544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:05.967612 32544 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:11:05.969913 32578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:05.971251 32544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:11:05.971391 32544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "4fd7de94815844e58b18d1961e838ea7"
format_stamp: "Formatted at 2026-09-26 20:11:04 on dist-test-slave-7034"
server_key: "eb2e9b1f0ec8b36fae92edd89a81548c"
server_key_iv: "0397c8d42a6ca66e272580a95ea437be"
server_key_version: "encryptionkey@0"
I20260926 20:11:05.971951 32544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:06.015800 32544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:06.019169 32544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:06.019416 32544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:06.020102 32544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:06.020961 32544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:06.021011 32544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:06.021052 32544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:06.021101 32544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:06.031038 32544 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:35747
I20260926 20:11:06.031065 32691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:35747 every 8 connection(s)
I20260926 20:11:06.032099 32544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:11:06.041683 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 32544
I20260926 20:11:06.042269 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:42533
--local_ip_for_outbound_sockets=127.23.50.130
--tserver_master_addrs=127.23.50.190:42825
--webserver_port=33185
--webserver_interface=127.23.50.130
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.50.130
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:06.045748 32695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:06.034481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:34323)
0926 20:11:06.034927 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:06.034978 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:06.036173 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:06.037284 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:06.037317 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:06.037671 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:06.038210 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.038225 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.039465 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.039468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.040347 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.040357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.041118 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.041121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.041252 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.042062 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:06.042084 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:06.044917 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":136,"threads_started":1}
W20260926 20:11:06.046214 32694 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:06.034431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:34323 (local address 127.23.50.190:42825)
0926 20:11:06.034882 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:06.034898 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:06.035122 (+ 224us) server_negotiation.cc:407] Connection header received
0926 20:11:06.036586 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:06.036598 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:06.036909 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:06.037201 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:06.038372 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.039343 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.040471 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.040973 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.041096 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.045240 (+ 4144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:34323: BlockingRecv error: recv got EOF from 127.23.50.129:34323 (error 108)
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":123,"threads_started":1}
W20260926 20:11:06.046397 32692 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:42825 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.50.190:42825: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:06.053423 32695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:06.046976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:56895)
0926 20:11:06.047159 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:06.047177 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:06.047274 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:06.047542 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:06.047547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:06.047741 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:06.047897 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.047908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.048659 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.048665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.049729 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.049746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.050228 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.050234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.050362 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.050971 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:06.050992 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:06.053240 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 20:11:06.053445 32694 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:06.047104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:56895 (local address 127.23.50.190:42825)
0926 20:11:06.047233 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:06.047236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:06.047249 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:11:06.047381 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:06.047383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:06.047424 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:06.047491 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:06.048027 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.048547 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.049896 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.050106 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.050183 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.053358 (+ 3175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:56895: BlockingRecv error: recv got EOF from 127.23.50.129:56895 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:06.059022 32695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:06.054147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:53023)
0926 20:11:06.054375 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:06.054387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:06.054468 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:06.054713 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:06.054715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:06.054871 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:06.054935 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.054940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.055641 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.055646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.056161 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.056169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.056589 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.056592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.056670 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.057131 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:06.057151 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:06.058878 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166}
W20260926 20:11:06.059098 32694 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:06.054257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:53023 (local address 127.23.50.190:42825)
0926 20:11:06.054355 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:06.054359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:06.054429 (+ 70us) server_negotiation.cc:407] Connection header received
0926 20:11:06.054570 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:06.054573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:06.054618 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:06.054697 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:06.055040 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.055534 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.056275 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.056486 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.056590 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.058985 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:53023: BlockingRecv error: recv got EOF from 127.23.50.129:53023 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 20:11:06.059340 32692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:11:06.159147 32696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:06.159415 32696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:06.159492 32696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:06.163493 32696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:06.163544 32696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:06.163636 32696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:11:06.168915 32696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.130.keytab
--principal=oryx/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:42533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=33185
--tserver_master_addrs=127.23.50.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:06.169972 32696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:06.171027 32696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:06.180562 32701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:06.180572 32702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:06.180572 32704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:06.180930 32696 server_base.cc:1034] running on GCE node
I20260926 20:11:06.181308 32696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:06.182029 32696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:06.183228 32696 hybrid_clock.cc:648] HybridClock initialized: now 1790453466183173 us; error 67 us; skew 500 ppm
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453466, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:06.186056 32696 init.cc:377] Logged in from keytab as oryx/127.23.50.130@KRBTEST.COM (short username oryx)
I20260926 20:11:06.186967 32696 webserver.cc:460] Webserver started at http://127.23.50.130:33185/ using document root <none> and password file <none>
I20260926 20:11:06.187575 32696 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:06.187698 32696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:06.191879 32696 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:11:06.194007 32711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:06.195327 32696 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:11:06.195477 32696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "45e24c5cad6342e4bedc34f6bf28eb19"
format_stamp: "Formatted at 2026-09-26 20:11:05 on dist-test-slave-7034"
server_key: "0761909a7aa71da572a995c1d3f756fb"
server_key_iv: "42e44ffcd09648d8e150f6641c61819a"
server_key_version: "encryptionkey@0"
I20260926 20:11:06.196050 32696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:06.207612 32696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:06.210896 32696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:06.211118 32696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:06.211726 32696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:06.212571 32696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:06.212646 32696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:06.212720 32696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:06.212764 32696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:06.222329 32696 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:42533
I20260926 20:11:06.222350 356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:42533 every 8 connection(s)
I20260926 20:11:06.223500 32696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:11:06.229630 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 32696
I20260926 20:11:06.230248 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:35795
--local_ip_for_outbound_sockets=127.23.50.131
--tserver_master_addrs=127.23.50.190:42825
--webserver_port=37425
--webserver_interface=127.23.50.131
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.50.131
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:11:06.236626 32694 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:06.225571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:36233 (local address 127.23.50.190:42825)
0926 20:11:06.225737 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:06.225741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:06.226739 (+ 998us) server_negotiation.cc:407] Connection header received
0926 20:11:06.228473 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:06.228476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:06.228578 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:06.228711 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:06.230046 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.230891 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.232187 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.232421 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.232545 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.236516 (+ 3971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:36233: BlockingRecv error: recv got EOF from 127.23.50.130:36233 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:11:06.237646 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:06.226131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:36233)
0926 20:11:06.226583 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:06.226626 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:06.228239 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:06.228915 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:06.228932 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:06.229312 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:06.229844 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.229890 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.231033 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.231036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.232053 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.232063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.232587 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.232592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.232780 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.233484 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:06.233508 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:06.236387 (+ 2879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
W20260926 20:11:06.238214 357 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:42825 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.50.190:42825: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:06.243963 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:06.238706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:46473)
0926 20:11:06.238832 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:06.238847 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:06.238935 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:06.239182 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:06.239185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:06.239368 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:06.239462 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.239467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.240209 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.240215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.240779 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.240787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.241255 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.241258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.241352 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.241937 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:06.241949 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:06.243814 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:11:06.244058 32694 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:06.238811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:46473 (local address 127.23.50.190:42825)
0926 20:11:06.238934 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:06.238940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:06.238956 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:11:06.239026 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:06.239029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:06.239080 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:06.239166 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:06.239578 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.240097 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.240907 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.241139 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.241260 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.243936 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:46473: BlockingRecv error: recv got EOF from 127.23.50.130:46473 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:06.249770 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:06.244491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:39029)
0926 20:11:06.244615 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:06.244630 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:06.244712 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:06.244943 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:06.244945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:06.245105 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:06.245197 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.245202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.245900 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.245904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.246781 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.246791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.247194 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.247198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.247307 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.247778 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:06.247792 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:06.249641 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 20:11:06.249996 357 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:11:06.250154 32694 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:06.244551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:39029 (local address 127.23.50.190:42825)
0926 20:11:06.244653 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:06.244656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:06.244669 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:11:06.244803 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:06.244805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:06.244840 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:06.244903 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:06.245301 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.245781 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.246888 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.247088 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.247166 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.249764 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:39029: BlockingRecv error: recv got EOF from 127.23.50.130:39029 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 20:11:06.352340 360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:06.352583 360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:06.352661 360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:06.356844 360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:06.356909 360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:06.357002 360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:11:06.362507 360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/krb5kdc/oryx_127.23.50.131.keytab
--principal=oryx/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:35795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=37425
--tserver_master_addrs=127.23.50.190:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:06.363555 360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:06.364564 360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:06.373963 365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:06.374034 366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:06.373966 368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:06.374054 360 server_base.cc:1034] running on GCE node
I20260926 20:11:06.374559 360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:06.375217 360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:06.376410 360 hybrid_clock.cc:648] HybridClock initialized: now 1790453466376390 us; error 33 us; skew 500 ppm
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453466, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:06.379210 360 init.cc:377] Logged in from keytab as oryx/127.23.50.131@KRBTEST.COM (short username oryx)
I20260926 20:11:06.380157 360 webserver.cc:460] Webserver started at http://127.23.50.131:37425/ using document root <none> and password file <none>
I20260926 20:11:06.380717 360 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:06.380792 360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:06.384652 360 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:11:06.386884 375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:06.388188 360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:11:06.388351 360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "ebc24cd550b545fca375d39d6426d0ba"
format_stamp: "Formatted at 2026-09-26 20:11:05 on dist-test-slave-7034"
server_key: "93508f201a497a805f330db578407a8a"
server_key_iv: "847564b6e99984cd3edf0d7541389d52"
server_key_version: "encryptionkey@0"
I20260926 20:11:06.388883 360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:06.401743 360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:06.404841 360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:06.405052 360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:06.405645 360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:06.406525 360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:06.406596 360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:06.406668 360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:06.406723 360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:06.416149 360 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:35795
I20260926 20:11:06.416206 488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:35795 every 8 connection(s)
I20260926 20:11:06.417193 360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:06.427351 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 360
W20260926 20:11:06.428287 32694 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:06.419024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:41709 (local address 127.23.50.190:42825)
0926 20:11:06.419173 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:06.419178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:06.420055 (+ 877us) server_negotiation.cc:407] Connection header received
0926 20:11:06.421336 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:06.421338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:06.421385 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:06.421468 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:06.422631 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.423153 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.424258 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.424516 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.424632 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.428156 (+ 3524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:41709: BlockingRecv error: recv got EOF from 127.23.50.131:41709 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:11:06.429420 491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:06.419478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:41709)
0926 20:11:06.419900 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:06.419941 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:06.421115 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:06.421626 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:06.421633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:06.421997 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:06.422497 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.422510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.423278 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.423281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.424129 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.424137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.424651 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.424654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.424787 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.425488 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:06.425504 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:06.428473 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
W20260926 20:11:06.430114 489 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:42825 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.50.190:42825: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:06.436071 491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:06.430543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:49933)
0926 20:11:06.430742 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:06.430753 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:06.430818 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:06.431030 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:06.431032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:06.431154 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:06.431236 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.431240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.432617 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.432620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.433149 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.433156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.433633 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.433637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.433728 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.434211 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:06.434222 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:06.435927 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128}
W20260926 20:11:06.436179 493 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:06.430659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:49933 (local address 127.23.50.190:42825)
0926 20:11:06.430795 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:06.430800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:06.430814 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:11:06.430881 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:06.430883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:06.430922 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:06.431008 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:06.431780 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.432502 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.433277 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.433521 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.433631 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.436044 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:49933: BlockingRecv error: recv got EOF from 127.23.50.131:49933 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:11:06.437395 32694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:06.428357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.23.50.190:42825)
0926 20:11:06.428594 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:06.428598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:06.428611 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:11:06.428648 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:06.428650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:06.428689 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:06.428781 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:06.429644 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.430229 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.431349 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.431563 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.431670 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.433120 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:06.433125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:06.433141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:06.433166 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:06.435168 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:06.435612 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:06.435621 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:06.435632 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:06.435752 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:06.436061 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:06.436066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:06.436067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:06.436520 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:06.436725 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:06.437023 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 20:11:06.437273 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":94,"threads_started":1}
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:06 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:06.442293 491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:06.436850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:43525)
0926 20:11:06.437006 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:06.437023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:06.437102 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:06.437388 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:06.437391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:06.437562 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:06.437643 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.437648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.438574 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.438578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.439345 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:06.439354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.439854 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.439858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:06.439945 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.440429 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:06.440443 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:06.442154 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260926 20:11:06.442385 493 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:06.436897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:43525 (local address 127.23.50.190:42825)
0926 20:11:06.437023 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:06.437027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:06.437064 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:11:06.437213 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:06.437218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:06.437261 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:06.437343 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:06.437754 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.438463 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.439464 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:06.439714 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:06.439821 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:06.442268 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:43525: BlockingRecv error: recv got EOF from 127.23.50.131:43525 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 20:11:06.442662 489 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 20:11:07 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:07 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:07.066723 494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:07.060074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:45133)
0926 20:11:07.060458 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:07.060477 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:07.060608 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:07.061106 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:07.061109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:07.061351 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:07.061472 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:07.061478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:07.062719 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:07.062723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:07.063601 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:07.063612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:07.064132 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:07.064136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:07.064212 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:07.064682 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:07.064693 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:07.066587 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
W20260926 20:11:07.066915 495 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:07.060539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:45133 (local address 127.23.50.190:42825)
0926 20:11:07.060792 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:07.060800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:07.060817 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:11:07.060861 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:07.060864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:07.060936 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:07.061055 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:07.061603 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:07.062584 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:07.063750 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:07.063973 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:07.064050 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:07.066761 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:45133: BlockingRecv error: recv got EOF from 127.23.50.129:45133 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
Sep 26 20:11:07 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:07 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:07.256233 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:07.250673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:40221)
0926 20:11:07.250978 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:07.250990 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:07.251121 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:07.251397 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:07.251399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:07.251618 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:07.251710 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:07.251715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:07.252530 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:07.252535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:07.253106 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:07.253114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:07.253573 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:07.253576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:07.253663 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:07.254154 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:07.254166 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:07.256048 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
W20260926 20:11:07.256295 495 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:07.250723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:40221 (local address 127.23.50.190:42825)
0926 20:11:07.250905 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:07.250909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:07.251062 (+ 153us) server_negotiation.cc:407] Connection header received
0926 20:11:07.251239 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:07.251241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:07.251286 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:07.251376 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:07.251838 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:07.252424 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:07.253227 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:07.253460 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:07.253567 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:07.256180 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:40221: BlockingRecv error: recv got EOF from 127.23.50.130:40221 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 20:11:07 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:07 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:07.449536 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:07.443384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:48683)
0926 20:11:07.443695 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:07.443709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:07.443817 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:07.444092 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:07.444095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:07.444314 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:07.444420 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:07.444425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:07.445213 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:07.445216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:07.445834 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:07.445843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:07.446498 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:07.446502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:07.446596 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:07.447204 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:07.447220 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:07.449348 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":95,"threads_started":1}
W20260926 20:11:07.449592 495 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:07.443443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:48683 (local address 127.23.50.190:42825)
0926 20:11:07.443586 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:07.443590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:07.443767 (+ 177us) server_negotiation.cc:407] Connection header received
0926 20:11:07.443896 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:07.443899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:07.443944 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:07.444012 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:07.444535 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:07.445087 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:07.446020 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:07.446320 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:07.446440 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:07.449476 (+ 3036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:48683: BlockingRecv error: recv got EOF from 127.23.50.131:48683 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 20:11:08 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:08 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:08.073117 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:08.067613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:41307)
0926 20:11:08.067892 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:08.067908 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:08.068015 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:08.068388 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:08.068390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:08.068566 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:08.068663 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:08.068668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:08.069431 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:08.069434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:08.070027 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:08.070037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:08.070540 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:08.070543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:08.070605 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:08.071181 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:08.071193 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:08.072977 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
W20260926 20:11:08.073354 499 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:08.067664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:41307 (local address 127.23.50.190:42825)
0926 20:11:08.067945 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:08.067948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:08.067960 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:11:08.068163 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:08.068167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:08.068232 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:08.068322 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:08.068791 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:08.069325 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:08.070200 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:08.070437 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:08.070524 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:08.073199 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:41307: BlockingRecv error: recv got EOF from 127.23.50.129:41307 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":108,"threads_started":1}
Sep 26 20:11:08 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:08 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:08.263554 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:08.257305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:46029)
0926 20:11:08.257680 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:08.257697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:08.257882 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:08.258299 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:08.258302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:08.258535 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:08.258633 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:08.258640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:08.259588 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:08.259594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:08.260292 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:08.260302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:08.260777 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:08.260781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:08.260869 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:08.261374 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:08.261387 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:08.263402 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":105,"threads_started":1}
W20260926 20:11:08.263751 499 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:08.257531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:46029 (local address 127.23.50.190:42825)
0926 20:11:08.257681 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:08.257687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:08.257794 (+ 107us) server_negotiation.cc:407] Connection header received
0926 20:11:08.258002 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:08.258005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:08.258131 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:08.258275 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:08.258772 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:08.259470 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:08.260420 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:08.260650 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:08.260761 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:08.263592 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:46029: BlockingRecv error: recv got EOF from 127.23.50.130:46029 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 20:11:08 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:08 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:08.460777 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:08.450719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:45081)
0926 20:11:08.451026 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:08.451040 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:08.451151 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:08.451467 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:08.451470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:08.451710 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:08.451880 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:08.451888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:08.452745 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:08.452748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:08.453405 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:08.453416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:08.454001 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:08.454005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:08.454137 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:08.458247 (+ 4110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:08.458264 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:08.460610 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
W20260926 20:11:08.460963 499 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:08.450824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:45081 (local address 127.23.50.190:42825)
0926 20:11:08.450981 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:08.450987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:08.451132 (+ 145us) server_negotiation.cc:407] Connection header received
0926 20:11:08.451279 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:08.451281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:08.451329 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:08.451397 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:08.452062 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:08.452624 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:08.453550 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:08.453772 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:08.453901 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:08.460821 (+ 6920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:45081: BlockingRecv error: recv got EOF from 127.23.50.131:45081 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 20:11:09 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:09 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:09.079455 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:09.073930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:45677)
0926 20:11:09.074201 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:09.074214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:09.074345 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:09.074636 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:09.074638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:09.074818 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:09.074916 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:09.074921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:09.075668 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:09.075671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:09.076373 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:09.076391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:09.076794 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:09.076797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:09.076860 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:09.077450 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:09.077462 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:09.079334 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":100,"threads_started":1}
W20260926 20:11:09.079648 503 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:09.074024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:45677 (local address 127.23.50.190:42825)
0926 20:11:09.074258 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:09.074261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:09.074293 (+ 32us) server_negotiation.cc:407] Connection header received
0926 20:11:09.074464 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:09.074466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:09.074519 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:09.074606 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:09.075025 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:09.075571 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:09.076471 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:09.076692 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:09.076772 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:09.079491 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:45677: BlockingRecv error: recv got EOF from 127.23.50.129:45677 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
Sep 26 20:11:09 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:09 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:09.270767 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:09.264642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:41039)
0926 20:11:09.265046 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:09.265063 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:09.265240 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:09.265539 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:09.265542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:09.265762 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:09.265933 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:09.265942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:09.266697 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:09.266702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:09.267272 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:09.267280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:09.267762 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:09.267766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:09.267851 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:09.268382 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:09.268394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:09.270529 (+ 2135us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":84,"threads_started":1}
W20260926 20:11:09.270841 503 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:09.264794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:41039 (local address 127.23.50.190:42825)
0926 20:11:09.264945 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:09.264949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:09.265196 (+ 247us) server_negotiation.cc:407] Connection header received
0926 20:11:09.265359 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:09.265362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:09.265421 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:09.265513 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:09.266036 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:09.266573 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:09.267401 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:09.267643 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:09.267755 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:09.270711 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:41039: BlockingRecv error: recv got EOF from 127.23.50.130:41039 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 20:11:09 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:09 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:09.467396 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:09.461676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:46291)
0926 20:11:09.462017 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:09.462030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:09.462131 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:09.462405 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:09.462407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:09.462583 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:09.462682 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:09.462687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:09.463446 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:09.463449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:09.464017 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:09.464026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:09.464507 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:09.464510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:09.464584 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:09.465123 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:09.465136 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:09.467256 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":127,"threads_started":1}
W20260926 20:11:09.467536 503 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:09.461820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:46291 (local address 127.23.50.190:42825)
0926 20:11:09.462018 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:09.462021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:09.462081 (+ 60us) server_negotiation.cc:407] Connection header received
0926 20:11:09.462239 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:09.462242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:09.462289 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:09.462365 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:09.462803 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:09.463341 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:09.464124 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:09.464369 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:09.464452 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:09.467413 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:46291: BlockingRecv error: recv got EOF from 127.23.50.131:46291 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 26 20:11:10 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:10 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:10.085613 506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:10.080270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:38179)
0926 20:11:10.080538 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:10.080553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:10.080652 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:10.080880 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:10.080882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:10.081055 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:10.081149 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:10.081154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:10.081964 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:10.081966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:10.082526 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:10.082534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:10.082945 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:10.082947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:10.083006 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:10.083487 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:10.083497 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:10.085479 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
W20260926 20:11:10.085835 507 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:10.080419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:38179 (local address 127.23.50.190:42825)
0926 20:11:10.080657 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:10.080661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:10.080675 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:11:10.080715 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:10.080717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:10.080758 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:10.080844 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:10.081254 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:10.081795 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:10.082628 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:10.082849 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:10.082926 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:10.085652 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:38179: BlockingRecv error: recv got EOF from 127.23.50.129:38179 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":84,"threads_started":1}
Sep 26 20:11:10 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:10 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:10.277930 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:10.271761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:49039)
0926 20:11:10.272130 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:10.272145 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:10.272302 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:10.272604 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:10.272607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:10.272827 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:10.272925 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:10.272931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:10.273757 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:10.273762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:10.274414 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:10.274423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:10.274939 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:10.274942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:10.275019 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:10.275586 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:10.275598 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:10.277753 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":93,"threads_started":1}
W20260926 20:11:10.278079 507 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:10.271903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:49039 (local address 127.23.50.190:42825)
0926 20:11:10.272096 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:10.272101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:10.272245 (+ 144us) server_negotiation.cc:407] Connection header received
0926 20:11:10.272418 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:10.272423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:10.272479 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:10.272577 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:10.273049 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:10.273635 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:10.274535 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:10.274821 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:10.274937 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:10.277959 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:49039: BlockingRecv error: recv got EOF from 127.23.50.130:49039 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 20:11:10 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:10 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:10.474098 509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:10.468394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:55383)
0926 20:11:10.468707 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:10.468721 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:10.468822 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:10.469126 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:10.469128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:10.469307 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:10.469407 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:10.469413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:10.470308 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:10.470312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:10.470873 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:10.470881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:10.471369 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:10.471373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:10.471463 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:10.472024 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:10.472036 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:10.473924 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
W20260926 20:11:10.474164 507 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:10.468582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:55383 (local address 127.23.50.190:42825)
0926 20:11:10.468760 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:10.468764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:10.468784 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:11:10.468916 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:10.468918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:10.468966 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:10.469040 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:10.469548 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:10.470179 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:10.471000 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:10.471224 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:10.471364 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:10.474052 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:55383: BlockingRecv error: recv got EOF from 127.23.50.131:55383 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 20:11:11 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:11 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:11.092051 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:11.086511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:54895)
0926 20:11:11.086780 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:11.086793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:11.086883 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:11.087104 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:11.087106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:11.087291 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:11.087385 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:11.087391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:11.088222 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:11.088225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:11.088785 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:11.088792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:11.089279 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:11.089282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:11.089350 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:11.089918 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:11.089932 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:11.091886 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
W20260926 20:11:11.092166 511 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:11.086581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:54895 (local address 127.23.50.190:42825)
0926 20:11:11.086880 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:11.086884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:11.086899 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:11:11.086948 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:11.086950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:11.086993 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:11.087077 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:11.087496 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:11.088111 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:11.088909 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:11.089144 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:11.089236 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:11.092030 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:54895: BlockingRecv error: recv got EOF from 127.23.50.129:54895 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":79,"threads_started":1}
Sep 26 20:11:11 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:11 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:11.284094 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:11.278782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:57151)
0926 20:11:11.279090 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:11.279104 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:11.279196 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:11.279412 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:11.279414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:11.279592 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:11.279684 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:11.279689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:11.280461 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:11.280463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:11.281026 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:11.281033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:11.281483 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:11.281486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:11.281547 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:11.282167 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:11.282179 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:11.283976 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":91,"threads_started":1}
W20260926 20:11:11.284308 511 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:11.278944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:57151 (local address 127.23.50.190:42825)
0926 20:11:11.279112 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:11.279117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:11.279155 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:11:11.279283 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:11.279285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:11.279329 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:11.279397 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:11.279811 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:11.280359 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:11.281140 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:11.281382 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:11.281465 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:11.284141 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:57151: BlockingRecv error: recv got EOF from 127.23.50.130:57151 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 20:11:11 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:11 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:11.481120 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:11.475078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:49635)
0926 20:11:11.475455 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:11.475467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:11.475617 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:11.475905 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:11.475908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:11.476186 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:11.476303 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:11.476309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:11.477091 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:11.477097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:11.477696 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:11.477704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:11.478238 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:11.478242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:11.478339 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:11.479001 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:11.479014 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:11.481003 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":147,"threads_started":1}
W20260926 20:11:11.481313 511 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:11.475196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:49635 (local address 127.23.50.190:42825)
0926 20:11:11.475388 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:11.475394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:11.475552 (+ 158us) server_negotiation.cc:407] Connection header received
0926 20:11:11.475726 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:11.475730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:11.475793 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:11.475885 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:11.476428 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:11.476973 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:11.477830 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:11.478111 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:11.478231 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:11.481146 (+ 2915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:49635: BlockingRecv error: recv got EOF from 127.23.50.131:49635 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 20:11:12 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:12 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:12.099395 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:12.093074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:54897)
0926 20:11:12.093446 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:12.093462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:12.093575 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:12.093892 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:12.093899 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:12.094136 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:12.094283 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:12.094291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:12.095122 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:12.095126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:12.095715 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:12.095725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:12.096211 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:12.096215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:12.096292 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:12.096821 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:12.096835 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:12.099180 (+ 2345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":106,"threads_started":1}
W20260926 20:11:12.099409 515 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:12.093143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:54897 (local address 127.23.50.190:42825)
0926 20:11:12.093457 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:12.093469 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:12.093542 (+ 73us) server_negotiation.cc:407] Connection header received
0926 20:11:12.093659 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:12.093661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:12.093706 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:12.093801 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:12.094403 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:12.094946 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:12.095833 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:12.096059 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:12.096141 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:12.099297 (+ 3156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:54897: BlockingRecv error: recv got EOF from 127.23.50.129:54897 (error 108)
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":117,"threads_started":1}
Sep 26 20:11:12 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:12 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:12.290414 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:12.284889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:33259)
0926 20:11:12.285165 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:12.285178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:12.285279 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:12.285515 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:12.285517 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:12.285702 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:12.285794 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:12.285799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:12.286622 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:12.286625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:12.287225 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:12.287233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:12.287798 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:12.287802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:12.287903 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:12.288421 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:12.288433 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:12.290297 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":93,"threads_started":1}
W20260926 20:11:12.290596 515 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:12.284951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:33259 (local address 127.23.50.190:42825)
0926 20:11:12.285108 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:12.285112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:12.285247 (+ 135us) server_negotiation.cc:407] Connection header received
0926 20:11:12.285362 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:12.285364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:12.285408 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:12.285477 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:12.285975 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:12.286523 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:12.287355 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:12.287576 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:12.287658 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:12.290450 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:33259: BlockingRecv error: recv got EOF from 127.23.50.130:33259 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 20:11:12 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:12 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:12.488300 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:12.482109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:39929)
0926 20:11:12.482576 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:12.482589 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:12.482695 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:12.483001 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:12.483007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:12.483262 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:12.483382 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:12.483389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:12.484364 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:12.484367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:12.484989 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:12.484999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:12.485484 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:12.485490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:12.485566 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:12.486265 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:12.486284 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:12.488144 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":116,"threads_started":1}
W20260926 20:11:12.488500 515 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:12.482269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:39929 (local address 127.23.50.190:42825)
0926 20:11:12.482437 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:12.482441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:12.482689 (+ 248us) server_negotiation.cc:407] Connection header received
0926 20:11:12.482823 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:12.482825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:12.482875 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:12.482950 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:12.483504 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:12.484088 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:12.485126 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:12.485360 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:12.485450 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:12.488323 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:39929: BlockingRecv error: recv got EOF from 127.23.50.131:39929 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 20:11:13 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:13 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:13.106482 518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:13.100289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:45157)
0926 20:11:13.100678 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:13.100693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:13.100780 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:13.101137 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:13.101144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:13.101379 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:13.101522 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:13.101529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:13.102531 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:13.102534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:13.103196 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:13.103208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:13.103717 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:13.103720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:13.103793 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:13.104298 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:13.104310 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:13.106295 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":104,"threads_started":1}
W20260926 20:11:13.106545 519 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:13.100462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:45157 (local address 127.23.50.190:42825)
0926 20:11:13.100738 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:13.100743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:13.100757 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:11:13.100909 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:13.100913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:13.100976 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:13.101089 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:13.101662 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:13.102355 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:13.103322 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:13.103558 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:13.103642 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:13.106422 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:45157: BlockingRecv error: recv got EOF from 127.23.50.129:45157 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":58,"threads_started":1}
Sep 26 20:11:13 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:13 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:13.297382 520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:13.291353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:58843)
0926 20:11:13.291748 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:13.291761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:13.291909 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:13.292257 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:13.292259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:13.292549 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:13.292682 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:13.292697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:13.293532 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:13.293537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:13.294258 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:13.294275 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:13.294737 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:13.294740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:13.294819 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:13.295368 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:13.295380 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:13.297202 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":108,"threads_started":1}
W20260926 20:11:13.297439 519 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:13.291479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:58843 (local address 127.23.50.190:42825)
0926 20:11:13.291663 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:13.291689 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:13.291844 (+ 155us) server_negotiation.cc:407] Connection header received
0926 20:11:13.292023 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:13.292031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:13.292119 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:13.292214 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:13.292812 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:13.293406 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:13.294389 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:13.294622 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:13.294729 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:13.297331 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:58843: BlockingRecv error: recv got EOF from 127.23.50.130:58843 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 20:11:13 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:13 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:13.510282 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:13.489212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:36327)
0926 20:11:13.489547 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:13.489560 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:13.489706 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:13.490025 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:13.490027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:13.490206 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:13.490305 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:13.490310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:13.491118 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:13.491121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:13.491760 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:13.491791 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:13.492372 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:13.492375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:13.492440 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:13.508163 (+ 15723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:13.508179 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:13.510160 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":84,"threads_started":1}
W20260926 20:11:13.510509 519 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:13.489271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:36327 (local address 127.23.50.190:42825)
0926 20:11:13.489441 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:13.489446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:13.489693 (+ 247us) server_negotiation.cc:407] Connection header received
0926 20:11:13.489800 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:13.489802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:13.489919 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:13.490002 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:13.490436 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:13.491005 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:13.491985 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:13.492246 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:13.492344 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:13.510378 (+ 18034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:36327: BlockingRecv error: recv got EOF from 127.23.50.131:36327 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 20:11:14 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:14 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:14.113515 522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:14.107391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:57317)
0926 20:11:14.107695 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:14.107709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:14.107800 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:14.108028 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:14.108030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:14.108251 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:14.108351 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:14.108357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:14.109453 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:14.109458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:14.110356 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:14.110370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:14.110885 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:14.110888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:14.110964 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:14.111453 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:14.111464 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:14.113396 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":122,"threads_started":1}
W20260926 20:11:14.113714 523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:14.107573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:57317 (local address 127.23.50.190:42825)
0926 20:11:14.107806 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:14.107809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:14.107824 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:11:14.107863 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:14.107865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:14.107906 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:14.107991 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:14.108533 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:14.109256 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:14.110486 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:14.110747 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:14.110835 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:14.113542 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:57317: BlockingRecv error: recv got EOF from 127.23.50.129:57317 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":85,"threads_started":1}
Sep 26 20:11:14 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:14 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:14.304914 524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:14.298305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:47153)
0926 20:11:14.298749 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:14.298788 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:14.298937 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:14.299500 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:14.299504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:14.299767 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:14.299906 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:14.299913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:14.300983 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:14.300989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:14.301564 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:14.301572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:14.302224 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:14.302228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:14.302309 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:14.302843 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:14.302855 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:14.304736 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":88,"threads_started":1}
W20260926 20:11:14.304977 523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:14.298455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:47153 (local address 127.23.50.190:42825)
0926 20:11:14.298721 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:14.298725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:14.298878 (+ 153us) server_negotiation.cc:407] Connection header received
0926 20:11:14.299047 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:14.299052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:14.299130 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:14.299233 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:14.300048 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:14.300872 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:14.301799 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:14.302100 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:14.302249 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:14.304866 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:47153: BlockingRecv error: recv got EOF from 127.23.50.130:47153 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
Sep 26 20:11:14 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:14 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:14.517210 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:14.511163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:56949)
0926 20:11:14.511505 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:14.511520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:14.511652 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:14.511929 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:14.511932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:14.512123 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:14.512230 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:14.512237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:14.513224 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:14.513229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:14.514067 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:14.514078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:14.514617 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:14.514620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:14.514695 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:14.515181 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:14.515192 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:14.517026 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":85,"threads_started":1}
W20260926 20:11:14.517314 523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:14.511374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:56949 (local address 127.23.50.190:42825)
0926 20:11:14.511531 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:14.511535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:14.511578 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:11:14.511731 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:14.511733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:14.511779 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:14.511851 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:14.512368 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:14.513045 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:14.514224 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:14.514515 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:14.514597 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:14.517159 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:56949: BlockingRecv error: recv got EOF from 127.23.50.131:56949 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 20:11:15 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:15 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:15.120105 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:15.114324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:46657)
0926 20:11:15.114656 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:15.114676 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:15.114770 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:15.115082 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:15.115085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:15.115288 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:15.115397 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:15.115404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:15.116201 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:15.116204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:15.116809 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:15.116818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:15.117311 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:15.117314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:15.117395 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:15.118012 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:15.118025 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:15.119924 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
W20260926 20:11:15.120347 527 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:15.114477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:46657 (local address 127.23.50.190:42825)
0926 20:11:15.114736 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:15.114739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:15.114753 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:11:15.114882 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:15.114885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:15.114928 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:15.115003 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:15.115539 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:15.116076 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:15.116939 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:15.117166 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:15.117250 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:15.120186 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:46657: BlockingRecv error: recv got EOF from 127.23.50.129:46657 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":92,"threads_started":1}
Sep 26 20:11:15 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:15 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:15.311941 528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:15.305809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:46919)
0926 20:11:15.306152 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:15.306169 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:15.306285 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:15.306648 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:15.306651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:15.306832 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:15.306939 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:15.306946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:15.307858 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:15.307862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:15.308623 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:15.308635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:15.309114 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:15.309118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:15.309203 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:15.309743 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:15.309755 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:15.311773 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":117,"threads_started":1}
W20260926 20:11:15.312043 527 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:15.305975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:46919 (local address 127.23.50.190:42825)
0926 20:11:15.306170 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:15.306174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:15.306251 (+ 77us) server_negotiation.cc:407] Connection header received
0926 20:11:15.306430 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:15.306432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:15.306483 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:15.306561 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:15.307077 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:15.307732 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:15.308747 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:15.308993 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:15.309107 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:15.311929 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:46919: BlockingRecv error: recv got EOF from 127.23.50.130:46919 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 20:11:15 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:15 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:15.523940 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:15.518169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:60633)
0926 20:11:15.518470 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:15.518484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:15.518585 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:15.518832 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:15.518833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:15.519013 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:15.519112 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:15.519117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:15.519881 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:15.519884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:15.520495 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:15.520504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:15.520973 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:15.520976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:15.521046 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:15.521613 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:15.521626 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:15.523826 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":86,"threads_started":1}
W20260926 20:11:15.524112 527 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:15.518255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:60633 (local address 127.23.50.190:42825)
0926 20:11:15.518402 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:15.518407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:15.518555 (+ 148us) server_negotiation.cc:407] Connection header received
0926 20:11:15.518668 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:15.518671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:15.518718 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:15.518812 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:15.519212 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:15.519739 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:15.520610 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:15.520848 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:15.520928 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:15.523956 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:60633: BlockingRecv error: recv got EOF from 127.23.50.131:60633 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 20:11:16 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:16 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:16.126904 530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:16.121146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:60021)
0926 20:11:16.121446 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:16.121459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:16.121553 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:16.121818 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:16.121821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:16.122162 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:16.122270 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:16.122276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:16.123084 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:16.123087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:16.123736 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:16.123746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:16.124279 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:16.124281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:16.124344 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:16.124922 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:16.124934 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:16.126783 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":78,"threads_started":1}
W20260926 20:11:16.127094 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:16.121192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:60021 (local address 127.23.50.190:42825)
0926 20:11:16.121465 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:16.121468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:16.121514 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:11:16.121619 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:16.121621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:16.121664 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:16.121742 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:16.122398 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:16.122956 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:16.123887 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:16.124132 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:16.124259 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:16.126948 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:60021: BlockingRecv error: recv got EOF from 127.23.50.129:60021 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":74,"threads_started":1}
Sep 26 20:11:16 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:16 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:16.318534 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:16.312800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:59625)
0926 20:11:16.313100 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:16.313117 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:16.313228 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:16.313472 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:16.313474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:16.313687 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:16.313786 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:16.313791 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:16.314656 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:16.314659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:16.315302 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:16.315311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:16.315745 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:16.315748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:16.315819 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:16.316379 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:16.316391 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:16.318394 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":69,"threads_started":1}
W20260926 20:11:16.318651 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:16.312883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:59625 (local address 127.23.50.190:42825)
0926 20:11:16.313087 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:16.313091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:16.313202 (+ 111us) server_negotiation.cc:407] Connection header received
0926 20:11:16.313312 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:16.313314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:16.313358 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:16.313424 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:16.313991 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:16.314533 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:16.315415 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:16.315633 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:16.315712 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:16.318533 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:59625: BlockingRecv error: recv got EOF from 127.23.50.130:59625 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 20:11:16 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:16 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:16.530491 533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:16.524789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:34841)
0926 20:11:16.525112 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:16.525130 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:16.525240 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:16.525528 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:16.525531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:16.525744 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:16.525867 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:16.525874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:16.526634 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:16.526639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:16.527214 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:16.527221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:16.527694 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:16.527698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:16.527781 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:16.528346 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:16.528358 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:16.530350 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
W20260926 20:11:16.530611 531 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:16.524965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:34841 (local address 127.23.50.190:42825)
0926 20:11:16.525112 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:16.525117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:16.525189 (+ 72us) server_negotiation.cc:407] Connection header received
0926 20:11:16.525358 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:16.525362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:16.525416 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:16.525501 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:16.525991 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:16.526521 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:16.527342 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:16.527580 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:16.527689 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:16.530494 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:34841: BlockingRecv error: recv got EOF from 127.23.50.131:34841 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 20:11:17 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:17 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:17.133836 534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:17.127823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:48069)
0926 20:11:17.128164 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:17.128178 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:17.128273 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:17.128543 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:17.128545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:17.128724 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:17.128844 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:17.128850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:17.129649 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:17.129651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:17.130252 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:17.130261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:17.130751 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:17.130753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:17.130830 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:17.131377 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:17.131389 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:17.133696 (+ 2307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
W20260926 20:11:17.134065 535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:17.127964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:48069 (local address 127.23.50.190:42825)
0926 20:11:17.128226 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:17.128230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:17.128254 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:11:17.128378 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:17.128380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:17.128426 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:17.128505 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:17.128977 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:17.129540 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:17.130404 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:17.130636 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:17.130728 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:17.133903 (+ 3175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:48069: BlockingRecv error: recv got EOF from 127.23.50.129:48069 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":62,"threads_started":1}
Sep 26 20:11:17 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:17 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:17.325287 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:17.319409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:35023)
0926 20:11:17.319755 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:17.319783 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:17.319929 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:17.320214 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:17.320217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:17.320432 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:17.320525 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:17.320531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:17.321355 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:17.321361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:17.321975 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:17.321984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:17.322559 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:17.322563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:17.322651 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:17.323277 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:17.323295 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:17.325163 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":100,"threads_started":1}
W20260926 20:11:17.325474 535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:17.319498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:35023 (local address 127.23.50.190:42825)
0926 20:11:17.319720 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:17.319725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:17.319869 (+ 144us) server_negotiation.cc:407] Connection header received
0926 20:11:17.320037 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:17.320042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:17.320090 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:17.320191 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:17.320654 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:17.321207 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:17.322105 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:17.322433 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:17.322553 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:17.325337 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:35023: BlockingRecv error: recv got EOF from 127.23.50.130:35023 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 26 20:11:17 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:17 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:17.537529 537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:17.531416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:52905)
0926 20:11:17.531700 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:17.531714 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:17.531818 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:17.532134 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:17.532137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:17.532342 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:17.532447 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:17.532452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:17.533252 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:17.533255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:17.533829 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:17.533838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:17.534355 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:17.534358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:17.534426 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:17.535055 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:17.535071 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:17.537407 (+ 2336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
W20260926 20:11:17.537734 535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:17.531530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:52905 (local address 127.23.50.190:42825)
0926 20:11:17.531711 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:17.531715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:17.531788 (+ 73us) server_negotiation.cc:407] Connection header received
0926 20:11:17.531906 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:17.531909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:17.531965 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:17.532048 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:17.532595 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:17.533128 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:17.533972 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:17.534200 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:17.534296 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:17.537558 (+ 3262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:52905: BlockingRecv error: recv got EOF from 127.23.50.131:52905 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 20:11:18 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:18 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
W20260926 20:11:18.140466 539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:18.134946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:54737 (local address 127.23.50.190:42825)
0926 20:11:18.135225 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:18.135229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:18.135243 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:11:18.135359 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:18.135361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:18.135403 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:18.135479 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:18.135942 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:18.136506 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:18.137294 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:18.137547 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:18.137639 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:18.140356 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:54737: BlockingRecv error: recv got EOF from 127.23.50.129:54737 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260926 20:11:18.140512 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:18.134879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:54737)
0926 20:11:18.135165 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:18.135180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:18.135272 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:18.135518 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:18.135520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:18.135698 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:18.135820 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:18.135826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:18.136600 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:18.136602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:18.137162 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:18.137170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:18.137658 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:18.137660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:18.137724 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:18.138334 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:18.138351 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:18.140159 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":69,"threads_started":1}
Sep 26 20:11:18 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:18 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:18.331900 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:18.326167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:56807)
0926 20:11:18.326474 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:18.326488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:18.326584 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:18.326851 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:18.326853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:18.327055 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:18.327159 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:18.327165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:18.327914 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:18.327917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:18.328490 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:18.328499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:18.328962 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:18.328964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:18.329028 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:18.329590 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:18.329602 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:18.331721 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":67,"threads_started":1}
W20260926 20:11:18.332312 539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:18.326366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:56807 (local address 127.23.50.190:42825)
0926 20:11:18.326543 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:18.326546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:18.326560 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:11:18.326676 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:18.326678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:18.326740 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:18.326811 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:18.327267 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:18.327795 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:18.328629 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:18.328854 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:18.328943 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:18.332225 (+ 3282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:56807: BlockingRecv error: recv got EOF from 127.23.50.130:56807 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 20:11:18 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:18 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:18.550956 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:18.538445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:40837)
0926 20:11:18.538741 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:18.538754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:18.538854 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:18.539091 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:18.539093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:18.539289 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:18.539394 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:18.539400 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:18.540147 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:18.540150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:18.540726 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:18.540737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:18.541229 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:18.541233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:18.541301 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:18.548753 (+ 7452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:18.548768 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:18.550771 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":107,"threads_started":1}
W20260926 20:11:18.551119 539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:18.538573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:40837 (local address 127.23.50.190:42825)
0926 20:11:18.538723 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:18.538726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:18.538808 (+ 82us) server_negotiation.cc:407] Connection header received
0926 20:11:18.538920 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:18.538922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:18.538965 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:18.539036 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:18.539506 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:18.540039 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:18.540844 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:18.541069 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:18.541197 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:18.550993 (+ 9796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:40837: BlockingRecv error: recv got EOF from 127.23.50.131:40837 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 20:11:19 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:19 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:19.147073 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:19.141385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:34737)
0926 20:11:19.141672 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:19.141688 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:19.141780 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:19.142076 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:19.142079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:19.142267 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:19.142362 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:19.142367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:19.143131 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:19.143134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:19.143762 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:19.143772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:19.144215 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:19.144218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:19.144320 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:19.144797 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:19.144809 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:19.146899 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
W20260926 20:11:19.147274 543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:19.141479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:34737 (local address 127.23.50.190:42825)
0926 20:11:19.141715 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:19.141719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:19.141754 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:11:19.141914 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:19.141917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:19.141964 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:19.142042 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:19.142462 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:19.143012 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:19.143880 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:19.144112 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:19.144195 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:19.147018 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:34737: BlockingRecv error: recv got EOF from 127.23.50.129:34737 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":84,"threads_started":1}
Sep 26 20:11:19 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:19 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:19.338812 544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:19.332831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:53251)
0926 20:11:19.333172 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:19.333189 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:19.333311 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:19.333590 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:19.333593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:19.333820 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:19.333983 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:19.333990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:19.334779 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:19.334785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:19.335357 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:19.335364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:19.335819 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:19.335822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:19.335905 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:19.336449 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:19.336460 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:19.338693 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
W20260926 20:11:19.339027 543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:19.333007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:53251 (local address 127.23.50.190:42825)
0926 20:11:19.333174 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:19.333180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:19.333267 (+ 87us) server_negotiation.cc:407] Connection header received
0926 20:11:19.333425 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:19.333429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:19.333479 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:19.333565 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:19.334098 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:19.334666 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:19.335478 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:19.335705 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:19.335816 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:19.338900 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:53251: BlockingRecv error: recv got EOF from 127.23.50.130:53251 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 20:11:19 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:19 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:19.558110 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:19.551916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:34919)
0926 20:11:19.552203 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:19.552221 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:19.552359 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:19.552706 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:19.552709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:19.552979 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:19.553109 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:19.553117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:19.554092 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:19.554095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:19.554683 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:19.554691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:19.555143 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:19.555145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:19.555221 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:19.555808 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:19.555823 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:19.557908 (+ 2085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":88,"threads_started":1}
W20260926 20:11:19.558295 543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:19.552001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:34919 (local address 127.23.50.190:42825)
0926 20:11:19.552204 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:19.552210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:19.552326 (+ 116us) server_negotiation.cc:407] Connection header received
0926 20:11:19.552505 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:19.552508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:19.552571 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:19.552660 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:19.553363 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:19.553980 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:19.554815 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:19.555039 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:19.555122 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:19.558129 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:34919: BlockingRecv error: recv got EOF from 127.23.50.131:34919 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 26 20:11:20 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:20 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:20.153680 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:20.147882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:41833)
0926 20:11:20.148164 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:20.148183 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:20.148308 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:20.148662 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:20.148664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:20.148871 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:20.148973 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:20.148978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:20.149786 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:20.149789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:20.150496 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:20.150506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:20.150927 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:20.150930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:20.150988 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:20.151509 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:20.151527 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:20.153487 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
W20260926 20:11:20.153743 547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:20.148032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:41833 (local address 127.23.50.190:42825)
0926 20:11:20.148424 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:20.148428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:20.148440 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:11:20.148476 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:20.148477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:20.148517 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:20.148593 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:20.149088 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:20.149659 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:20.150599 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:20.150827 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:20.150908 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:20.153619 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:41833: BlockingRecv error: recv got EOF from 127.23.50.129:41833 (error 108)
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":94,"threads_started":1}
Sep 26 20:11:20 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:20 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:20.345055 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:20.339633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:41089)
0926 20:11:20.339931 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:20.339945 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:20.340040 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:20.340285 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:20.340287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:20.340472 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:20.340571 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:20.340575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:20.341378 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:20.341381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:20.342020 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:20.342030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:20.342512 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:20.342515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:20.342577 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:20.343089 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:20.343101 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:20.344917 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":117,"threads_started":1}
W20260926 20:11:20.345270 547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:20.339728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:41089 (local address 127.23.50.190:42825)
0926 20:11:20.339888 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:20.339891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:20.340015 (+ 124us) server_negotiation.cc:407] Connection header received
0926 20:11:20.340123 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:20.340125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:20.340171 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:20.340250 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:20.340715 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:20.341257 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:20.342147 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:20.342407 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:20.342496 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:20.345071 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:41089: BlockingRecv error: recv got EOF from 127.23.50.130:41089 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 20:11:20 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:20 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.131@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:20.565039 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:20.559061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.131:34083)
0926 20:11:20.559401 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:20.559414 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:20.559519 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:20.559739 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:20.559741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:20.559920 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:20.560016 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:20.560021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:20.560824 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:20.560827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:20.561507 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:20.561523 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:20.562030 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:20.562033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:20.562100 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:20.562700 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:20.562712 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:20.564844 (+ 2132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":90,"threads_started":1}
W20260926 20:11:20.565153 547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:20.559188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:34083 (local address 127.23.50.190:42825)
0926 20:11:20.559399 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:20.559402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:20.559486 (+ 84us) server_negotiation.cc:407] Connection header received
0926 20:11:20.559601 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:20.559604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:20.559653 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:20.559722 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:20.560151 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:20.560693 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:20.561644 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:20.561911 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:20.561994 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:20.565003 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.131:34083: BlockingRecv error: recv got EOF from 127.23.50.131:34083 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 20:11:21 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:21 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.129@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:21.160728 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:21.154928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.129:36883)
0926 20:11:21.155217 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:21.155235 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:21.155373 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:21.155728 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:21.155731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:21.155971 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:21.156116 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:21.156124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.157007 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.157010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:21.157598 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:21.157609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.158086 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.158089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:21.158160 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:21.158698 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:21.158711 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:21.160604 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
W20260926 20:11:21.160979 551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:21.155024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:36883 (local address 127.23.50.190:42825)
0926 20:11:21.155309 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:21.155315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:21.155331 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:11:21.155500 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:21.155504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:21.155567 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:21.155671 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:21.156246 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.156882 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.157725 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.157971 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.158050 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:21.160837 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.129:36883: BlockingRecv error: recv got EOF from 127.23.50.129:36883 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
Sep 26 20:11:21 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
Sep 26 20:11:21 dist-test-slave-7034 krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.50.130@KRBTEST.COM for oryx/127.23.50.190@KRBTEST.COM, Server not found in Kerberos database
I20260926 20:11:21.352959 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:21.346173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42825 (local address 127.23.50.130:48747)
0926 20:11:21.346470 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:21.346490 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:21.346632 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:21.346948 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:21.346951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:21.347241 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:21.347391 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:21.347414 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.348484 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.348491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:21.349459 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:21.349470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.349997 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.350001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:21.350084 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:21.350709 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:21.350727 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:21.352819 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.50.190:42825: Server oryx/127.23.50.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
W20260926 20:11:21.353127 551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:21.346237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:48747 (local address 127.23.50.190:42825)
0926 20:11:21.346386 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:21.346391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:21.346584 (+ 193us) server_negotiation.cc:407] Connection header received
0926 20:11:21.346733 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:21.346736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:21.346804 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:21.346899 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:21.347531 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.348288 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.349587 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.349885 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.349981 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:21.352996 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.50.130:48747: BlockingRecv error: recv got EOF from 127.23.50.130:48747 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:11:21.429605 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 32544
I20260926 20:11:21.435369 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 32696
I20260926 20:11:21.440656 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 360
I20260926 20:11:21.446115 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 32474
2026-09-26T20:11:21Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18122 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T20:11:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:11:21Z Disabled control of system clock
I20260926 20:11:21.477097 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:43607
--webserver_interface=127.23.50.190
--webserver_port=0
--builtin_ntp_servers=127.23.50.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:43607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 20:11:21.590955 556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:21.591251 556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:21.591288 556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:21.595463 556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:11:21.595534 556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:21.595551 556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:21.595567 556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:11:21.595583 556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:11:21.600438 556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:43607
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:43607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:21.601555 556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:21.603075 556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:21.612140 564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:21.612172 561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:21.612185 562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:21.612283 556 server_base.cc:1034] running on GCE node
I20260926 20:11:21.613075 556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:21.614261 556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:21.615432 556 hybrid_clock.cc:648] HybridClock initialized: now 1790453481615398 us; error 50 us; skew 500 ppm
I20260926 20:11:21.617385 556 webserver.cc:460] Webserver started at http://127.23.50.190:40621/ using document root <none> and password file <none>
I20260926 20:11:21.617992 556 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:21.618060 556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:21.618243 556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:21.620316 556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "3ee958d71b2f4fbb8afc14f5264603d5"
format_stamp: "Formatted at 2026-09-26 20:11:21 on dist-test-slave-7034"
server_key: "b5d6bd303bc09175d913ddd1b7503239"
server_key_iv: "72ca9887a5c911007f4f2a05ef5e38bd"
server_key_version: "encryptionkey@0"
I20260926 20:11:21.620867 556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "3ee958d71b2f4fbb8afc14f5264603d5"
format_stamp: "Formatted at 2026-09-26 20:11:21 on dist-test-slave-7034"
server_key: "b5d6bd303bc09175d913ddd1b7503239"
server_key_iv: "72ca9887a5c911007f4f2a05ef5e38bd"
server_key_version: "encryptionkey@0"
I20260926 20:11:21.624889 556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 20:11:21.627553 570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:21.628921 556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:11:21.629084 556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "3ee958d71b2f4fbb8afc14f5264603d5"
format_stamp: "Formatted at 2026-09-26 20:11:21 on dist-test-slave-7034"
server_key: "b5d6bd303bc09175d913ddd1b7503239"
server_key_iv: "72ca9887a5c911007f4f2a05ef5e38bd"
server_key_version: "encryptionkey@0"
I20260926 20:11:21.629215 556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:21.646139 556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:21.646903 556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:21.647122 556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:21.654359 556 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:43607
I20260926 20:11:21.654368 622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:43607 every 8 connection(s)
I20260926 20:11:21.655370 556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:11:21.658663 623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:21.663347 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 556
I20260926 20:11:21.663443 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:11:21.663755 23754 external_mini_cluster.cc:1428] Setting key 9ffc971a11eabb5ff339f7fb9d7a1813
I20260926 20:11:21.665627 623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5: Bootstrap starting.
I20260926 20:11:21.668154 623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:21.668959 623 log.cc:824] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:11:21.671120 623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5: No bootstrap required, opened a new log
I20260926 20:11:21.673720 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:21.665434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.23.50.190:43607)
0926 20:11:21.666245 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:21.666263 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:21.666301 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:11:21.666957 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:21.666975 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:21.667391 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:21.667738 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:21.668261 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.669366 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.670348 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.670830 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.671001 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:21.671895 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:21.671900 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:21.671914 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:11:21.672323 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:21.672383 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:11:21.672403 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:21.672517 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:21.672672 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 20:11:21.672947 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":151,"threads_started":1}
I20260926 20:11:21.675817 623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee958d71b2f4fbb8afc14f5264603d5" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43607 } }
I20260926 20:11:21.676141 623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:21.676235 623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee958d71b2f4fbb8afc14f5264603d5, State: Initialized, Role: FOLLOWER
I20260926 20:11:21.676715 623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee958d71b2f4fbb8afc14f5264603d5" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43607 } }
I20260926 20:11:21.676856 623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:11:21.676955 623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:11:21.677060 623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:21.678242 623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee958d71b2f4fbb8afc14f5264603d5" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43607 } }
I20260926 20:11:21.678645 623 leader_election.cc:302] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ee958d71b2f4fbb8afc14f5264603d5; no voters:
I20260926 20:11:21.679040 623 leader_election.cc:288] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:11:21.679368 628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:11:21.679620 628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3ee958d71b2f4fbb8afc14f5264603d5, State: Running, Role: LEADER
I20260926 20:11:21.679939 628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee958d71b2f4fbb8afc14f5264603d5" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43607 } }
I20260926 20:11:21.680450 623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:11:21.681767 629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ee958d71b2f4fbb8afc14f5264603d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee958d71b2f4fbb8afc14f5264603d5" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43607 } } }
I20260926 20:11:21.681941 629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:11:21.682377 630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ee958d71b2f4fbb8afc14f5264603d5. Latest consensus state: current_term: 1 leader_uuid: "3ee958d71b2f4fbb8afc14f5264603d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee958d71b2f4fbb8afc14f5264603d5" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43607 } } }
I20260926 20:11:21.682474 630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:11:21.683117 638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:11:21.686102 638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:11:21.691874 638 catalog_manager.cc:1347] Generated new cluster ID: a603ed3125474033ac94c3be4f21a9f2
I20260926 20:11:21.691957 638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:11:21.719208 638 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:11:21.720276 638 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:11:21.728403 638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ee958d71b2f4fbb8afc14f5264603d5: Generated new TSK 0
I20260926 20:11:21.729184 638 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:11:21.732743 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43607
--builtin_ntp_servers=127.23.50.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 20:11:21.851055 648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:21.851341 648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:21.851379 648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:21.856383 648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:21.856469 648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:21.856534 648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:11:21.862766 648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:21.864053 648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:21.865705 648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:21.876927 656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:21.877012 648 server_base.cc:1034] running on GCE node
W20260926 20:11:21.876927 653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:21.876994 654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:21.877610 648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:21.878343 648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:21.879516 648 hybrid_clock.cc:648] HybridClock initialized: now 1790453481879497 us; error 34 us; skew 500 ppm
I20260926 20:11:21.881687 648 webserver.cc:460] Webserver started at http://127.23.50.129:41381/ using document root <none> and password file <none>
I20260926 20:11:21.882315 648 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:21.882409 648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:21.882608 648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:21.884730 648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "7af891e5d51e46cab40c3481581be17b"
format_stamp: "Formatted at 2026-09-26 20:11:21 on dist-test-slave-7034"
server_key: "9c09f981b786fcded0ece090b4f6877e"
server_key_iv: "86ac3dc869f79c33e1c158e929481913"
server_key_version: "encryptionkey@0"
I20260926 20:11:21.885268 648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "7af891e5d51e46cab40c3481581be17b"
format_stamp: "Formatted at 2026-09-26 20:11:21 on dist-test-slave-7034"
server_key: "9c09f981b786fcded0ece090b4f6877e"
server_key_iv: "86ac3dc869f79c33e1c158e929481913"
server_key_version: "encryptionkey@0"
I20260926 20:11:21.889328 648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 20:11:21.892134 662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:21.893545 648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:11:21.893661 648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "7af891e5d51e46cab40c3481581be17b"
format_stamp: "Formatted at 2026-09-26 20:11:21 on dist-test-slave-7034"
server_key: "9c09f981b786fcded0ece090b4f6877e"
server_key_iv: "86ac3dc869f79c33e1c158e929481913"
server_key_version: "encryptionkey@0"
I20260926 20:11:21.893793 648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:21.920614 648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:21.921443 648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:21.921680 648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:21.922387 648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:21.923441 648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:21.923516 648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:21.923592 648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:21.923645 648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:21.933115 648 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:44385
I20260926 20:11:21.933143 775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:44385 every 8 connection(s)
I20260926 20:11:21.934180 648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:11:21.940987 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 648
I20260926 20:11:21.941115 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:11:21.941443 23754 external_mini_cluster.cc:1428] Setting key b623d3ab9dacd6f4fac6caba9edcad54
I20260926 20:11:21.943570 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43607
--builtin_ntp_servers=127.23.50.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:11:21.944857 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:21.936197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:33189 (local address 127.23.50.190:43607)
0926 20:11:21.936347 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:21.936351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:21.937601 (+ 1250us) server_negotiation.cc:407] Connection header received
0926 20:11:21.939179 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:21.939182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:21.939239 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:21.939329 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:21.940321 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.940851 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.942562 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.942823 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.942935 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:21.944220 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:21.944227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:21.944233 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:11:21.944252 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:21.944293 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:11:21.944306 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:21.944389 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:21.944565 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 20:11:21.944641 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:11:21.946373 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:21.936684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43607 (local address 127.23.50.129:33189)
0926 20:11:21.937426 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:21.937472 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:21.938948 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:21.939514 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:21.939521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:21.939614 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:21.940149 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:21.940162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.940990 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.940994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:21.942446 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:21.942458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:21.942988 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:21.942993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:21.943187 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:21.943935 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:11:21.943937 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:21.943982 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:11:21.944054 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:21.944407 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:21.944416 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:21.944421 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:11:21.944534 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 20:11:21.944805 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":117,"threads_started":1}
I20260926 20:11:21.947803 776 heartbeater.cc:347] Connected to a master server at 127.23.50.190:43607
I20260926 20:11:21.948089 776 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:21.948679 776 heartbeater.cc:511] Master 127.23.50.190:43607 requested a full tablet report, sending...
I20260926 20:11:21.950606 587 ts_manager.cc:194] Registered new tserver with Master: 7af891e5d51e46cab40c3481581be17b (127.23.50.129:44385)
I20260926 20:11:21.952586 587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.50.129:33189
W20260926 20:11:22.063072 779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:22.063390 779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:22.063431 779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:22.067610 779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:22.067672 779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:22.067727 779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:11:22.072847 779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:22.074045 779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:22.075420 779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:22.088477 785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:22.088483 784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:22.088479 787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:22.089041 779 server_base.cc:1034] running on GCE node
I20260926 20:11:22.089601 779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:22.090549 779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:22.091724 779 hybrid_clock.cc:648] HybridClock initialized: now 1790453482091698 us; error 47 us; skew 500 ppm
I20260926 20:11:22.094415 779 webserver.cc:460] Webserver started at http://127.23.50.130:41961/ using document root <none> and password file <none>
I20260926 20:11:22.095110 779 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:22.095177 779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:22.095472 779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:22.098459 779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "07bd042d356d48a9b161b6ad9292d4a2"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "3c90a198f194bf28a8fe07d9524a5354"
server_key_iv: "8d267cd5f0bb7253722c357a313b9fd4"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.099205 779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "07bd042d356d48a9b161b6ad9292d4a2"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "3c90a198f194bf28a8fe07d9524a5354"
server_key_iv: "8d267cd5f0bb7253722c357a313b9fd4"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.104302 779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 20:11:22.107555 793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:22.109350 779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:11:22.109478 779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "07bd042d356d48a9b161b6ad9292d4a2"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "3c90a198f194bf28a8fe07d9524a5354"
server_key_iv: "8d267cd5f0bb7253722c357a313b9fd4"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.109633 779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:22.127303 779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:22.128036 779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:22.128252 779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:22.128877 779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:22.129935 779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:22.130005 779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:22.130087 779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:22.130122 779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:22.139561 779 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:33311
I20260926 20:11:22.139582 906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:33311 every 8 connection(s)
I20260926 20:11:22.140585 779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:11:22.141322 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 779
I20260926 20:11:22.141469 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:11:22.141757 23754 external_mini_cluster.cc:1428] Setting key 16ba8bb2dbbe950282d42df37860797e
I20260926 20:11:22.144032 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43607
--builtin_ntp_servers=127.23.50.148:44853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 20:11:22.151036 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:22.144026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:33643 (local address 127.23.50.190:43607)
0926 20:11:22.144145 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:22.144149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:22.144169 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:11:22.146134 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:22.146141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:22.146208 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:22.146324 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:22.147259 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:22.147793 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:22.148983 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:22.149223 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:22.149334 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:22.150513 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:22.150517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:22.150519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:11:22.150533 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:22.150561 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:11:22.150570 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:22.150646 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:22.150825 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:11:22.150890 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:11:22.151938 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:22.143043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43607 (local address 127.23.50.130:33643)
0926 20:11:22.143908 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:22.143966 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:22.145904 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:22.146488 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:22.146495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:22.146585 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:22.147104 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:22.147116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:22.147922 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:22.147933 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:22.148850 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:22.148860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:22.149384 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:22.149388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:22.149556 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:22.150240 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:11:22.150242 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:22.150283 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:11:22.150353 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:22.150664 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:22.150681 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:22.150686 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:11:22.150825 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 20:11:22.151127 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":160,"threads_started":1}
I20260926 20:11:22.153323 907 heartbeater.cc:347] Connected to a master server at 127.23.50.190:43607
I20260926 20:11:22.153577 907 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:22.154179 907 heartbeater.cc:511] Master 127.23.50.190:43607 requested a full tablet report, sending...
I20260926 20:11:22.155364 587 ts_manager.cc:194] Registered new tserver with Master: 07bd042d356d48a9b161b6ad9292d4a2 (127.23.50.130:33311)
I20260926 20:11:22.156143 587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.50.130:33643
W20260926 20:11:22.261343 910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:22.261626 910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:22.261663 910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:22.265909 910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:22.265969 910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:22.266026 910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:11:22.270929 910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:22.271996 910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:22.273366 910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:22.283478 916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:22.283491 918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:22.283480 915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:22.283749 910 server_base.cc:1034] running on GCE node
I20260926 20:11:22.284200 910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:22.284886 910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:22.286044 910 hybrid_clock.cc:648] HybridClock initialized: now 1790453482286027 us; error 29 us; skew 500 ppm
I20260926 20:11:22.287971 910 webserver.cc:460] Webserver started at http://127.23.50.131:38169/ using document root <none> and password file <none>
I20260926 20:11:22.288628 910 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:22.288697 910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:22.288872 910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:22.291085 910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "18aad8d56d5a43449f07274fed9df183"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "e9ba098ba7d90e9edc3c68758379b0ff"
server_key_iv: "7ea0fc68831a61ec8997e0196da20638"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.291610 910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "18aad8d56d5a43449f07274fed9df183"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "e9ba098ba7d90e9edc3c68758379b0ff"
server_key_iv: "7ea0fc68831a61ec8997e0196da20638"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.295439 910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:11:22.298161 924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:22.299491 910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:11:22.299643 910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "18aad8d56d5a43449f07274fed9df183"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "e9ba098ba7d90e9edc3c68758379b0ff"
server_key_iv: "7ea0fc68831a61ec8997e0196da20638"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.299772 910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:22.327257 910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:22.328050 910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:22.328307 910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:22.328922 910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:22.329900 910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:22.329972 910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:22.330046 910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:22.330099 910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:22.339632 910 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:45173
I20260926 20:11:22.339659 1037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:45173 every 8 connection(s)
I20260926 20:11:22.340754 910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:11:22.341207 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 910
I20260926 20:11:22.341324 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:11:22.341647 23754 external_mini_cluster.cc:1428] Setting key c39023a18df324b4f616425fa9539ad5
I20260926 20:11:22.350683 626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:22.342943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:47405 (local address 127.23.50.190:43607)
0926 20:11:22.343095 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:22.343100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:22.344423 (+ 1323us) server_negotiation.cc:407] Connection header received
0926 20:11:22.345835 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:22.345855 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:22.345916 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:22.345998 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:22.346934 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:22.347482 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:22.348678 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:22.348915 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:22.349027 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:22.350173 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:22.350178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:22.350180 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:11:22.350197 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:22.350223 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:11:22.350231 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:22.350305 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:22.350474 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 20:11:22.350538 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:11:22.351606 1040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:22.343534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43607 (local address 127.23.50.131:47405)
0926 20:11:22.344276 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:22.344318 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:22.345605 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:22.346175 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:22.346182 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:22.346272 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:22.346785 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:22.346798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:22.347615 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:22.347621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:22.348544 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:22.348554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:22.349073 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:22.349077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:22.349249 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:22.349907 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:11:22.349908 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:22.349952 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:11:22.350017 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:22.350319 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:22.350330 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:22.350337 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:11:22.350470 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 20:11:22.350782 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":121,"threads_started":1}
I20260926 20:11:22.352783 1038 heartbeater.cc:347] Connected to a master server at 127.23.50.190:43607
I20260926 20:11:22.353027 1038 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:22.353551 1038 heartbeater.cc:511] Master 127.23.50.190:43607 requested a full tablet report, sending...
I20260926 20:11:22.354885 587 ts_manager.cc:194] Registered new tserver with Master: 18aad8d56d5a43449f07274fed9df183 (127.23.50.131:45173)
I20260926 20:11:22.355679 587 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.50.131:47405
I20260926 20:11:22.356181 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:11:22.359119 626 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:22.358191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.23.50.190:43607)
0926 20:11:22.358347 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:22.358352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:22.358559 (+ 207us) server_negotiation.cc:407] Connection header received
0926 20:11:22.358646 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:22.358648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:22.358686 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:22.358753 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:22.358998 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35966: BlockingRecv error: recv got EOF from 127.0.0.1:35966 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:11:22.359236 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 648
I20260926 20:11:22.364676 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 779
I20260926 20:11:22.370244 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 910
I20260926 20:11:22.375924 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 556
2026-09-26T20:11:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (929 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T20:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:11:22Z Disabled control of system clock
I20260926 20:11:22.406657 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:35531
--webserver_interface=127.23.50.190
--webserver_port=0
--builtin_ntp_servers=127.23.50.148:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:35531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:11:22.527159 1049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:22.527454 1049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:22.527493 1049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:22.531479 1049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:11:22.531539 1049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:22.531555 1049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:22.531574 1049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:11:22.531589 1049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:11:22.536481 1049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:35531
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:35531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:22.537561 1049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:22.538956 1049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:22.548547 1057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:22.548561 1054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:22.548557 1055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:22.548827 1049 server_base.cc:1034] running on GCE node
I20260926 20:11:22.549427 1049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:22.550611 1049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:22.551789 1049 hybrid_clock.cc:648] HybridClock initialized: now 1790453482551753 us; error 49 us; skew 500 ppm
I20260926 20:11:22.553930 1049 webserver.cc:460] Webserver started at http://127.23.50.190:34963/ using document root <none> and password file <none>
I20260926 20:11:22.554540 1049 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:22.554606 1049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:22.554872 1049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:22.557050 1049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "bdcae5365fae48a8b7c48d3d9679e756"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "9101f5487a08704a1c79c54e13d63426"
server_key_iv: "d25acdb71ef7a66f79d318447c1ec46d"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.557646 1049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "bdcae5365fae48a8b7c48d3d9679e756"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "9101f5487a08704a1c79c54e13d63426"
server_key_iv: "d25acdb71ef7a66f79d318447c1ec46d"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.561626 1049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:11:22.564384 1063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:22.565712 1049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:11:22.565924 1049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "bdcae5365fae48a8b7c48d3d9679e756"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "9101f5487a08704a1c79c54e13d63426"
server_key_iv: "d25acdb71ef7a66f79d318447c1ec46d"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.566098 1049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:22.570837 1049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:22.571477 1049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:22.571671 1049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:22.579103 1049 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:35531
I20260926 20:11:22.579133 1115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:35531 every 8 connection(s)
I20260926 20:11:22.580101 1049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:11:22.582783 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 1049
I20260926 20:11:22.582916 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:11:22.583197 23754 external_mini_cluster.cc:1428] Setting key bb2bdf6250225a603653ef6439fc1e0c
I20260926 20:11:22.583704 1116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:22.589325 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:22.584587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.23.50.190:35531)
0926 20:11:22.585267 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:22.585283 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:22.585320 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:11:22.586443 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:22.586463 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:22.586836 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:22.587126 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:22.587971 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:22.587976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:22.587989 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:11:22.587998 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:22.588053 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:11:22.588070 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:22.588196 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:22.588327 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 20:11:22.588607 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":145,"threads_started":1}
I20260926 20:11:22.590772 1116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756: Bootstrap starting.
I20260926 20:11:22.592921 1116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:22.593660 1116 log.cc:824] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756: Log is configured to *not* fsync() on all Append() calls
I20260926 20:11:22.595679 1116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756: No bootstrap required, opened a new log
I20260926 20:11:22.599284 1116 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcae5365fae48a8b7c48d3d9679e756" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35531 } }
I20260926 20:11:22.599534 1116 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:22.599597 1116 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcae5365fae48a8b7c48d3d9679e756, State: Initialized, Role: FOLLOWER
I20260926 20:11:22.600028 1116 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcae5365fae48a8b7c48d3d9679e756" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35531 } }
I20260926 20:11:22.600154 1116 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:11:22.600237 1116 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:11:22.600340 1116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:22.601284 1116 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcae5365fae48a8b7c48d3d9679e756" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35531 } }
I20260926 20:11:22.601634 1116 leader_election.cc:302] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdcae5365fae48a8b7c48d3d9679e756; no voters:
I20260926 20:11:22.602008 1116 leader_election.cc:288] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:11:22.602149 1121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:11:22.602399 1121 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [term 1 LEADER]: Becoming Leader. State: Replica: bdcae5365fae48a8b7c48d3d9679e756, State: Running, Role: LEADER
I20260926 20:11:22.602734 1121 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcae5365fae48a8b7c48d3d9679e756" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35531 } }
I20260926 20:11:22.603408 1116 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:11:22.604502 1123 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdcae5365fae48a8b7c48d3d9679e756. Latest consensus state: current_term: 1 leader_uuid: "bdcae5365fae48a8b7c48d3d9679e756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcae5365fae48a8b7c48d3d9679e756" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35531 } } }
I20260926 20:11:22.604635 1123 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [sys.catalog]: This master's current role is: LEADER
I20260926 20:11:22.604950 1122 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdcae5365fae48a8b7c48d3d9679e756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdcae5365fae48a8b7c48d3d9679e756" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35531 } } }
I20260926 20:11:22.605057 1122 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756 [sys.catalog]: This master's current role is: LEADER
I20260926 20:11:22.605149 1130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:11:22.608350 1130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:11:22.614382 1130 catalog_manager.cc:1347] Generated new cluster ID: 016811a3c09b48abbe0037baac1bf780
I20260926 20:11:22.614475 1130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:11:22.627978 1130 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:11:22.628338 1130 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:11:22.637416 1130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdcae5365fae48a8b7c48d3d9679e756: Generated new TSK 0
I20260926 20:11:22.638388 1130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:11:22.651382 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:35531
--builtin_ntp_servers=127.23.50.148:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 20:11:22.769690 1140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:22.770040 1140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:22.770128 1140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:22.774297 1140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:22.774359 1140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:22.774420 1140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:11:22.779388 1140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:22.780505 1140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:22.781863 1140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:22.791719 1145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:22.791719 1146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:22.791857 1140 server_base.cc:1034] running on GCE node
W20260926 20:11:22.792106 1148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:22.792519 1140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:22.793166 1140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:22.794359 1140 hybrid_clock.cc:648] HybridClock initialized: now 1790453482794344 us; error 29 us; skew 500 ppm
I20260926 20:11:22.796203 1140 webserver.cc:460] Webserver started at http://127.23.50.129:41539/ using document root <none> and password file <none>
I20260926 20:11:22.796826 1140 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:22.796921 1140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:22.797158 1140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:22.799404 1140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "1a01a5213d6443e7b365b56ca2213c37"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "c8261246a24691148099321b5ddeff3d"
server_key_iv: "9f3943228cb7bf74e118d89646b39f59"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.800019 1140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "1a01a5213d6443e7b365b56ca2213c37"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "c8261246a24691148099321b5ddeff3d"
server_key_iv: "9f3943228cb7bf74e118d89646b39f59"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.803984 1140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:11:22.806948 1154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:22.808331 1140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:11:22.808477 1140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "1a01a5213d6443e7b365b56ca2213c37"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "c8261246a24691148099321b5ddeff3d"
server_key_iv: "9f3943228cb7bf74e118d89646b39f59"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.808607 1140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:22.813477 1140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:22.814181 1140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:22.814370 1140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:22.814925 1140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:22.815872 1140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:22.815944 1140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:22.816011 1140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:22.816061 1140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:22.825172 1140 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:34517
I20260926 20:11:22.825186 1267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:34517 every 8 connection(s)
I20260926 20:11:22.826241 1140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:11:22.828540 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 1140
I20260926 20:11:22.828667 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:11:22.828962 23754 external_mini_cluster.cc:1428] Setting key e20c386c886cbb3eaab3183177f4d517
I20260926 20:11:22.831282 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:35531
--builtin_ntp_servers=127.23.50.148:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 20:11:22.833607 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:22.828288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:53539 (local address 127.23.50.190:35531)
0926 20:11:22.828441 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:22.828445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:22.829936 (+ 1491us) server_negotiation.cc:407] Connection header received
0926 20:11:22.831681 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:22.831688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:22.831738 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:22.831840 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:22.833137 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:22.833140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:22.833141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:11:22.833153 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:22.833209 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:11:22.833217 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:22.833294 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:22.833465 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 20:11:22.833504 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:11:22.834882 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:22.828889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:35531 (local address 127.23.50.129:53539)
0926 20:11:22.829748 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:22.829813 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:22.831413 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:22.832001 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:22.832008 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:22.832105 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:22.832804 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:11:22.832807 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:22.832870 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:11:22.832970 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:22.833305 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:22.833316 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:22.833325 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:11:22.833514 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 20:11:22.833800 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":178,"threads_started":1}
I20260926 20:11:22.836221 1268 heartbeater.cc:347] Connected to a master server at 127.23.50.190:35531
I20260926 20:11:22.836493 1268 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:22.837037 1268 heartbeater.cc:511] Master 127.23.50.190:35531 requested a full tablet report, sending...
I20260926 20:11:22.838721 1080 ts_manager.cc:194] Registered new tserver with Master: 1a01a5213d6443e7b365b56ca2213c37 (127.23.50.129:34517)
W20260926 20:11:22.945823 1271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:22.946111 1271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:22.946202 1271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:22.950332 1271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:22.950387 1271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:22.950487 1271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:11:22.955416 1271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:22.956538 1271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:22.957955 1271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:22.967767 1276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:22.967794 1277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:22.967794 1279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:22.968212 1271 server_base.cc:1034] running on GCE node
I20260926 20:11:22.968665 1271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:22.969425 1271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:22.970633 1271 hybrid_clock.cc:648] HybridClock initialized: now 1790453482970585 us; error 64 us; skew 500 ppm
I20260926 20:11:22.972666 1271 webserver.cc:460] Webserver started at http://127.23.50.130:33091/ using document root <none> and password file <none>
I20260926 20:11:22.973243 1271 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:22.973306 1271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:22.973475 1271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:22.975757 1271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "e363551a19e34fb9b5e1c66c85d96197"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "bc201ea27a0f20472beda37236f24b7f"
server_key_iv: "fc5b726b7e8ca4566a5445a08620e526"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.976334 1271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "e363551a19e34fb9b5e1c66c85d96197"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "bc201ea27a0f20472beda37236f24b7f"
server_key_iv: "fc5b726b7e8ca4566a5445a08620e526"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.980341 1271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 20:11:22.983035 1285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:22.984354 1271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:11:22.984496 1271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "e363551a19e34fb9b5e1c66c85d96197"
format_stamp: "Formatted at 2026-09-26 20:11:22 on dist-test-slave-7034"
server_key: "bc201ea27a0f20472beda37236f24b7f"
server_key_iv: "fc5b726b7e8ca4566a5445a08620e526"
server_key_version: "encryptionkey@0"
I20260926 20:11:22.984615 1271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:22.989615 1271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:22.990327 1271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:22.990518 1271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:22.991101 1271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:22.992060 1271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:22.992107 1271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:22.992175 1271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:22.992214 1271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:23.002072 1271 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:38355
I20260926 20:11:23.002089 1398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:38355 every 8 connection(s)
I20260926 20:11:23.003028 1271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:11:23.008534 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 1271
I20260926 20:11:23.008671 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:11:23.008989 23754 external_mini_cluster.cc:1428] Setting key 960a348850250a6d01c789581cd86155
I20260926 20:11:23.009565 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:23.005024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:60837 (local address 127.23.50.190:35531)
0926 20:11:23.005168 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:23.005172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:23.006402 (+ 1230us) server_negotiation.cc:407] Connection header received
0926 20:11:23.007667 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:23.007670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:23.007708 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:23.007787 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:23.009067 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:23.009072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:23.009077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:11:23.009093 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:23.009120 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:11:23.009130 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:23.009211 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:23.009366 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 20:11:23.009405 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:11:23.010623 1401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:23.005514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:35531 (local address 127.23.50.130:60837)
0926 20:11:23.006248 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:23.006292 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:23.007473 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:23.007960 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:23.007966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:23.008057 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:23.008732 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:11:23.008734 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:23.008790 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:11:23.008882 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:23.009212 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:23.009220 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:23.009227 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:11:23.009404 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 20:11:23.009649 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":130,"threads_started":1}
I20260926 20:11:23.011122 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:35531
--builtin_ntp_servers=127.23.50.148:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 20:11:23.011987 1399 heartbeater.cc:347] Connected to a master server at 127.23.50.190:35531
I20260926 20:11:23.012272 1399 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:23.012825 1399 heartbeater.cc:511] Master 127.23.50.190:35531 requested a full tablet report, sending...
I20260926 20:11:23.014007 1080 ts_manager.cc:194] Registered new tserver with Master: e363551a19e34fb9b5e1c66c85d96197 (127.23.50.130:38355)
W20260926 20:11:23.128739 1402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:23.129045 1402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:23.129086 1402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:23.133257 1402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:23.133323 1402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:23.133380 1402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:11:23.138392 1402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:35531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:23.139568 1402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:23.141021 1402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:23.151233 1410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:23.151257 1408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:23.151161 1407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:23.151669 1402 server_base.cc:1034] running on GCE node
I20260926 20:11:23.152124 1402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:23.152855 1402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:23.154013 1402 hybrid_clock.cc:648] HybridClock initialized: now 1790453483153996 us; error 29 us; skew 500 ppm
I20260926 20:11:23.156049 1402 webserver.cc:460] Webserver started at http://127.23.50.131:35471/ using document root <none> and password file <none>
I20260926 20:11:23.156697 1402 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:23.156760 1402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:23.156984 1402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:23.159314 1402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "c9920effcb0e4219828b2b8d9a065ee1"
format_stamp: "Formatted at 2026-09-26 20:11:23 on dist-test-slave-7034"
server_key: "b0324a1ceadc3154a3ebbc8ecd32a6d1"
server_key_iv: "63af544eb0bd99fc8005e89c998f053d"
server_key_version: "encryptionkey@0"
I20260926 20:11:23.159897 1402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "c9920effcb0e4219828b2b8d9a065ee1"
format_stamp: "Formatted at 2026-09-26 20:11:23 on dist-test-slave-7034"
server_key: "b0324a1ceadc3154a3ebbc8ecd32a6d1"
server_key_iv: "63af544eb0bd99fc8005e89c998f053d"
server_key_version: "encryptionkey@0"
I20260926 20:11:23.163774 1402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 20:11:23.166473 1416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:23.167824 1402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:11:23.167943 1402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "c9920effcb0e4219828b2b8d9a065ee1"
format_stamp: "Formatted at 2026-09-26 20:11:23 on dist-test-slave-7034"
server_key: "b0324a1ceadc3154a3ebbc8ecd32a6d1"
server_key_iv: "63af544eb0bd99fc8005e89c998f053d"
server_key_version: "encryptionkey@0"
I20260926 20:11:23.168051 1402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:23.172875 1402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:23.173488 1402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:23.173674 1402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:23.174269 1402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:23.175158 1402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:23.175203 1402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:23.175268 1402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:23.175304 1402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:23.184264 1402 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:46607
I20260926 20:11:23.184283 1529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:46607 every 8 connection(s)
I20260926 20:11:23.185225 1402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:11:23.187971 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 1402
I20260926 20:11:23.188105 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:11:23.188426 23754 external_mini_cluster.cc:1428] Setting key 9a186036c0f61b7e89c196a4e7188cfb
I20260926 20:11:23.192271 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:23.187233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:49959 (local address 127.23.50.190:35531)
0926 20:11:23.187386 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:23.187390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:23.188921 (+ 1531us) server_negotiation.cc:407] Connection header received
0926 20:11:23.190619 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:23.190622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:23.190673 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:23.190763 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:23.191915 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:23.191918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:23.191920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:11:23.191930 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:23.191948 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:11:23.191955 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:23.192023 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:23.192123 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 20:11:23.192160 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:11:23.193188 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:23.187786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:35531 (local address 127.23.50.131:49959)
0926 20:11:23.188712 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:23.188785 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:23.190358 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:23.190941 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:23.190949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:23.191043 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:23.191658 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:11:23.191660 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:23.191705 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:11:23.191784 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:23.192012 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:23.192016 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:23.192019 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 20:11:23.192142 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 20:11:23.192378 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":174,"threads_started":1}
I20260926 20:11:23.194379 1530 heartbeater.cc:347] Connected to a master server at 127.23.50.190:35531
I20260926 20:11:23.194608 1530 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:23.195040 1530 heartbeater.cc:511] Master 127.23.50.190:35531 requested a full tablet report, sending...
I20260926 20:11:23.196074 1080 ts_manager.cc:194] Registered new tserver with Master: c9920effcb0e4219828b2b8d9a065ee1 (127.23.50.131:46607)
I20260926 20:11:23.202463 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 20:11:23.205816 1119 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:11:23.204667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.23.50.190:35531)
0926 20:11:23.204796 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:23.204800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:23.205090 (+ 290us) server_negotiation.cc:407] Connection header received
0926 20:11:23.205259 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:23.205262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:23.205302 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:23.205417 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:23.205650 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43734: BlockingRecv error: recv got EOF from 127.0.0.1:43734 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:11:23.205981 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 1140
I20260926 20:11:23.212586 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 1271
I20260926 20:11:23.218156 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 1402
I20260926 20:11:23.224205 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 1049
2026-09-26T20:11:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (848 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:11:23 dist-test-slave-7034 krb5kdc[1540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:11:23 dist-test-slave-7034 krb5kdc[1540](info): set up 2 sockets
Sep 26 20:11:23 dist-test-slave-7034 krb5kdc[1540](info): commencing operation
krb5kdc: starting...
W20260926 20:11:24.263593 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:11:24 dist-test-slave-7034 krb5kdc[1540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:11:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:11:24.389232 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:44157
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:44157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:24.501542 1553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:24.501835 1553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:24.501963 1553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:24.506095 1553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:11:24.506155 1553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:24.506174 1553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:24.506191 1553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:11:24.506207 1553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:11:24.511581 1553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:44157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:44157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:24.512615 1553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:24.513592 1553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:24.522240 1561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:24.522280 1553 server_base.cc:1034] running on GCE node
W20260926 20:11:24.522243 1558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:24.522240 1559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:24.523070 1553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:24.524272 1553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:24.525437 1553 hybrid_clock.cc:648] HybridClock initialized: now 1790453484525425 us; error 30 us; skew 500 ppm
Sep 26 20:11:24 dist-test-slave-7034 krb5kdc[1540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:24.532255 1553 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:11:24.533178 1553 webserver.cc:460] Webserver started at http://127.23.50.190:36921/ using document root <none> and password file <none>
I20260926 20:11:24.533712 1553 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:24.533769 1553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:24.534022 1553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:24.536149 1553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "32af490af34c4ba58b4e1702653d7ef1"
format_stamp: "Formatted at 2026-09-26 20:11:24 on dist-test-slave-7034"
server_key: "29f438ecee839e39f27ef32254b0a937"
server_key_iv: "4dd4144f1e3d5eaf05569abb9d2d5596"
server_key_version: "encryptionkey@0"
I20260926 20:11:24.536679 1553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "32af490af34c4ba58b4e1702653d7ef1"
format_stamp: "Formatted at 2026-09-26 20:11:24 on dist-test-slave-7034"
server_key: "29f438ecee839e39f27ef32254b0a937"
server_key_iv: "4dd4144f1e3d5eaf05569abb9d2d5596"
server_key_version: "encryptionkey@0"
I20260926 20:11:24.540608 1553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:11:24.543200 1568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:24.544561 1553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:11:24.544718 1553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "32af490af34c4ba58b4e1702653d7ef1"
format_stamp: "Formatted at 2026-09-26 20:11:24 on dist-test-slave-7034"
server_key: "29f438ecee839e39f27ef32254b0a937"
server_key_iv: "4dd4144f1e3d5eaf05569abb9d2d5596"
server_key_version: "encryptionkey@0"
I20260926 20:11:24.544853 1553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:24.562525 1553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:24.565541 1553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:24.565761 1553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:24.573357 1553 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:44157
I20260926 20:11:24.573385 1620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:44157 every 8 connection(s)
I20260926 20:11:24.574407 1553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:11:24.575738 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 1553
I20260926 20:11:24.575860 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:11:24.576170 23754 external_mini_cluster.cc:1428] Setting key 03de12c6c4a9b413d854d9087e9a831d
I20260926 20:11:24.577976 1621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:24.583951 1621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1: Bootstrap starting.
Sep 26 20:11:24 dist-test-slave-7034 krb5kdc[1540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:11:24.586491 1621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:24.587381 1621 log.cc:824] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1: Log is configured to *not* fsync() on all Append() calls
I20260926 20:11:24.589377 1621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1: No bootstrap required, opened a new log
I20260926 20:11:24.591002 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:24.577641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.23.50.190:44157)
0926 20:11:24.578129 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:24.578145 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:24.578182 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:11:24.578843 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:24.578859 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:24.579188 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:24.579545 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:24.580433 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:24.581441 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:24.582409 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:24.582900 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:24.583075 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:24.585670 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:24.585675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:24.585688 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:24.585718 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:24.587816 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:24.588340 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:24.588345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:24.588350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:24.588464 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:24.588766 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:24.588769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:24.588771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:24.589185 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:24.589398 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:24.589689 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 20:11:24.590036 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":181,"threads_started":1}
I20260926 20:11:24.593365 1621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32af490af34c4ba58b4e1702653d7ef1" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 44157 } }
I20260926 20:11:24.593586 1621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:24.593622 1621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32af490af34c4ba58b4e1702653d7ef1, State: Initialized, Role: FOLLOWER
I20260926 20:11:24.594072 1621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32af490af34c4ba58b4e1702653d7ef1" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 44157 } }
I20260926 20:11:24.594199 1621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:11:24.594247 1621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:11:24.594318 1621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:24.595290 1621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32af490af34c4ba58b4e1702653d7ef1" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 44157 } }
I20260926 20:11:24.595649 1621 leader_election.cc:302] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32af490af34c4ba58b4e1702653d7ef1; no voters:
I20260926 20:11:24.596046 1621 leader_election.cc:288] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:11:24.596201 1626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:11:24.596446 1626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [term 1 LEADER]: Becoming Leader. State: Replica: 32af490af34c4ba58b4e1702653d7ef1, State: Running, Role: LEADER
I20260926 20:11:24.596755 1626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32af490af34c4ba58b4e1702653d7ef1" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 44157 } }
I20260926 20:11:24.597357 1621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:11:24.598507 1627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32af490af34c4ba58b4e1702653d7ef1. Latest consensus state: current_term: 1 leader_uuid: "32af490af34c4ba58b4e1702653d7ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32af490af34c4ba58b4e1702653d7ef1" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 44157 } } }
I20260926 20:11:24.598649 1627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [sys.catalog]: This master's current role is: LEADER
I20260926 20:11:24.598595 1628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32af490af34c4ba58b4e1702653d7ef1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32af490af34c4ba58b4e1702653d7ef1" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 44157 } } }
I20260926 20:11:24.598706 1628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1 [sys.catalog]: This master's current role is: LEADER
I20260926 20:11:24.599040 1635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:11:24.602375 1635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:11:24.608282 1635 catalog_manager.cc:1347] Generated new cluster ID: 04651ab9af8a496c8622f1cc8f969cc9
I20260926 20:11:24.608361 1635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:11:24.623522 1635 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:11:24.624614 1635 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:11:24.630539 1635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1: Generated new TSK 0
I20260926 20:11:24.631315 1635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:11:24.663290 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:44157
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:24.787358 1647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:24.787663 1647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:24.787734 1647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:24.792232 1647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:24.792322 1647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:24.792413 1647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:11:24.798353 1647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:44157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:24.799562 1647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:24.800642 1647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:24.810711 1655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:24.810773 1653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:24.810952 1647 server_base.cc:1034] running on GCE node
W20260926 20:11:24.811094 1652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:24.811620 1647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:24.812355 1647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:24.813567 1647 hybrid_clock.cc:648] HybridClock initialized: now 1790453484813548 us; error 34 us; skew 500 ppm
Sep 26 20:11:24 dist-test-slave-7034 krb5kdc[1540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:24.816609 1647 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:11:24.817633 1647 webserver.cc:460] Webserver started at http://127.23.50.129:34725/ using document root <none> and password file <none>
I20260926 20:11:24.818312 1647 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:24.818415 1647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:24.818667 1647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:24.820952 1647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "b5922a655cd14b9086a07e3d4a2e26e8"
format_stamp: "Formatted at 2026-09-26 20:11:24 on dist-test-slave-7034"
server_key: "56d391498ecc03a9b1eb5faf4b84d03a"
server_key_iv: "9509773ce49f2c8af0c0bea02ab95e9c"
server_key_version: "encryptionkey@0"
I20260926 20:11:24.821609 1647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "b5922a655cd14b9086a07e3d4a2e26e8"
format_stamp: "Formatted at 2026-09-26 20:11:24 on dist-test-slave-7034"
server_key: "56d391498ecc03a9b1eb5faf4b84d03a"
server_key_iv: "9509773ce49f2c8af0c0bea02ab95e9c"
server_key_version: "encryptionkey@0"
I20260926 20:11:24.825821 1647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 20:11:24.828760 1662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:24.830178 1647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 20:11:24.830358 1647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "b5922a655cd14b9086a07e3d4a2e26e8"
format_stamp: "Formatted at 2026-09-26 20:11:24 on dist-test-slave-7034"
server_key: "56d391498ecc03a9b1eb5faf4b84d03a"
server_key_iv: "9509773ce49f2c8af0c0bea02ab95e9c"
server_key_version: "encryptionkey@0"
I20260926 20:11:24.830490 1647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:24.847414 1647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:24.850947 1647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:24.851177 1647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:24.851850 1647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:24.852768 1647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:24.852849 1647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:24.852933 1647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:24.852983 1647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:24.863476 1647 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:45807
I20260926 20:11:24.863495 1775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:45807 every 8 connection(s)
I20260926 20:11:24.864463 1647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:11:24.870730 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 1647
I20260926 20:11:24.870858 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:11:24.871160 23754 external_mini_cluster.cc:1428] Setting key 7cf9bb63a4e629839bc1758561aefa10
Sep 26 20:11:24 dist-test-slave-7034 krb5kdc[1540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453484, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:11:24.879765 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:24.866382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:40033 (local address 127.23.50.190:44157)
0926 20:11:24.866536 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:24.866539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:24.867365 (+ 826us) server_negotiation.cc:407] Connection header received
0926 20:11:24.868823 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:24.868826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:24.868878 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:24.868952 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:24.870266 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:24.871032 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:24.872426 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:24.872718 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:24.872828 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:24.875765 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:24.875770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:24.875772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:24.875798 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:24.877243 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:24.877982 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:24.877985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:24.877986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:24.878116 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:24.878595 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:24.878598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:24.878600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:24.878782 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:24.878861 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:24.879539 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 20:11:24.879643 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:11:24.881197 1778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:24.866701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:44157 (local address 127.23.50.129:40033)
0926 20:11:24.867194 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:24.867243 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:24.868585 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:24.869133 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:24.869143 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:24.869511 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:24.870121 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:24.870137 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:24.871159 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:24.871162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:24.872327 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:24.872337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:24.872872 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:24.872875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:24.873023 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:24.873669 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:24.873686 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:24.875582 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:24.877374 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:24.877384 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:24.877403 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:24.877869 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:24.878231 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:24.878238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:24.878244 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:24.878459 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:24.878898 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:24.878905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:24.879251 (+ 346us) client_negotiation.cc:769] Sending connection context
0926 20:11:24.879494 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 20:11:24.879997 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:11:24.882666 1776 heartbeater.cc:347] Connected to a master server at 127.23.50.190:44157
I20260926 20:11:24.882954 1776 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:24.883517 1776 heartbeater.cc:511] Master 127.23.50.190:44157 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
I20260926 20:11:24.885324 1585 ts_manager.cc:194] Registered new tserver with Master: b5922a655cd14b9086a07e3d4a2e26e8 (127.23.50.129:45807)
I20260926 20:11:24.886631 1585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:40033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:11:24.901592 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:44157
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:25.016778 1781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:25.017117 1781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:25.017215 1781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:25.021415 1781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:25.021526 1781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:25.021625 1781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:11:25.027041 1781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:44157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:25.028189 1781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:25.029243 1781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:25.038951 1787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:25.038951 1789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:25.038941 1786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:25.039228 1781 server_base.cc:1034] running on GCE node
I20260926 20:11:25.039831 1781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:25.040525 1781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:25.041683 1781 hybrid_clock.cc:648] HybridClock initialized: now 1790453485041652 us; error 47 us; skew 500 ppm
Sep 26 20:11:25 dist-test-slave-7034 krb5kdc[1540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:25.044528 1781 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:11:25.045399 1781 webserver.cc:460] Webserver started at http://127.23.50.130:39263/ using document root <none> and password file <none>
I20260926 20:11:25.046027 1781 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:25.046119 1781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:25.046360 1781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:25.048511 1781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "1bf3e363a0314b61bcd2a0d6493ddc75"
format_stamp: "Formatted at 2026-09-26 20:11:25 on dist-test-slave-7034"
server_key: "cce201d40ed7207e4adf8e26ac0a5ee3"
server_key_iv: "a2d38dc2bba7fde5f2b13dc14df4a7fc"
server_key_version: "encryptionkey@0"
I20260926 20:11:25.049115 1781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "1bf3e363a0314b61bcd2a0d6493ddc75"
format_stamp: "Formatted at 2026-09-26 20:11:25 on dist-test-slave-7034"
server_key: "cce201d40ed7207e4adf8e26ac0a5ee3"
server_key_iv: "a2d38dc2bba7fde5f2b13dc14df4a7fc"
server_key_version: "encryptionkey@0"
I20260926 20:11:25.052920 1781 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:11:25.056458 1796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:25.058151 1781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:11:25.058285 1781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "1bf3e363a0314b61bcd2a0d6493ddc75"
format_stamp: "Formatted at 2026-09-26 20:11:25 on dist-test-slave-7034"
server_key: "cce201d40ed7207e4adf8e26ac0a5ee3"
server_key_iv: "a2d38dc2bba7fde5f2b13dc14df4a7fc"
server_key_version: "encryptionkey@0"
I20260926 20:11:25.058445 1781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:25.073172 1781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:25.076683 1781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:25.076846 1781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:25.077405 1781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:25.078271 1781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:25.078320 1781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:25.078361 1781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:25.078377 1781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:25.087491 1781 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:39589
I20260926 20:11:25.087513 1909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:39589 every 8 connection(s)
I20260926 20:11:25.088469 1781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:11:25 dist-test-slave-7034 krb5kdc[1540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:11:25.098659 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 1781
I20260926 20:11:25.098755 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:11:25.099084 23754 external_mini_cluster.cc:1428] Setting key e6c82bfe24fd0a5460f5a40c862074c9
I20260926 20:11:25.104162 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.090409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:50171 (local address 127.23.50.190:44157)
0926 20:11:25.090551 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:25.090560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:25.091379 (+ 819us) server_negotiation.cc:407] Connection header received
0926 20:11:25.092726 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:25.092728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:25.092775 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:25.092850 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:25.094089 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.094611 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.095765 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.096003 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.096096 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.098913 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:25.098918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:25.098922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:25.098963 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:25.100709 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:25.102502 (+ 1793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:25.102506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:25.102508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:25.102596 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:25.103046 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:25.103049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:25.103050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:25.103231 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:25.103351 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:25.103884 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 20:11:25.104005 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:11:25.105052 1912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.090786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:44157 (local address 127.23.50.130:50171)
0926 20:11:25.091218 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:25.091262 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:25.092521 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:25.093017 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:25.093024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:25.093388 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:25.093955 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.093972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.094724 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.094726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.095615 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.095625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.096144 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.096147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.096292 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.096908 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:25.096924 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:25.098687 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:25.101950 (+ 3263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:25.101960 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:25.101982 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:25.102397 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:25.102716 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:25.102720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:25.102722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:25.102929 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:25.103350 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:25.103358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:25.103625 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 20:11:25.103825 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 20:11:25.104174 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":113,"threads_started":1}
I20260926 20:11:25.106351 1910 heartbeater.cc:347] Connected to a master server at 127.23.50.190:44157
I20260926 20:11:25.106666 1910 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:25.107251 1910 heartbeater.cc:511] Master 127.23.50.190:44157 requested a full tablet report, sending...
I20260926 20:11:25.108422 1585 ts_manager.cc:194] Registered new tserver with Master: 1bf3e363a0314b61bcd2a0d6493ddc75 (127.23.50.130:39589)
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:11:25.109117 1585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:50171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:11:25.128746 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:44157
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:11:25.240217 1915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:11:25.240528 1915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:11:25.240621 1915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:11:25.244731 1915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:11:25.244831 1915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:11:25.244925 1915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:11:25.250531 1915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:44157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:11:25.251652 1915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:11:25.252707 1915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:11:25.262627 1920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:25.262794 1923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:11:25.262709 1921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:11:25.262921 1915 server_base.cc:1034] running on GCE node
I20260926 20:11:25.263458 1915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:11:25.264137 1915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:11:25.265302 1915 hybrid_clock.cc:648] HybridClock initialized: now 1790453485265283 us; error 31 us; skew 500 ppm
Sep 26 20:11:25 dist-test-slave-7034 krb5kdc[1540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:11:25.268141 1915 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:11:25.269002 1915 webserver.cc:460] Webserver started at http://127.23.50.131:38487/ using document root <none> and password file <none>
I20260926 20:11:25.269553 1915 fs_manager.cc:359] Metadata directory not provided
I20260926 20:11:25.269613 1915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:11:25.269896 1915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:11:25.271893 1915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b"
format_stamp: "Formatted at 2026-09-26 20:11:25 on dist-test-slave-7034"
server_key: "90856c0b0dd940175a8dfc0f87c46e69"
server_key_iv: "8c42cabdf4f422ce00b713d2e754c374"
server_key_version: "encryptionkey@0"
I20260926 20:11:25.272452 1915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b"
format_stamp: "Formatted at 2026-09-26 20:11:25 on dist-test-slave-7034"
server_key: "90856c0b0dd940175a8dfc0f87c46e69"
server_key_iv: "8c42cabdf4f422ce00b713d2e754c374"
server_key_version: "encryptionkey@0"
I20260926 20:11:25.276294 1915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:11:25.278831 1930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:25.280073 1915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:11:25.280251 1915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b"
format_stamp: "Formatted at 2026-09-26 20:11:25 on dist-test-slave-7034"
server_key: "90856c0b0dd940175a8dfc0f87c46e69"
server_key_iv: "8c42cabdf4f422ce00b713d2e754c374"
server_key_version: "encryptionkey@0"
I20260926 20:11:25.280354 1915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:11:25.307601 1915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:11:25.310582 1915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:11:25.310810 1915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:11:25.311431 1915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:11:25.312258 1915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:11:25.312304 1915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:25.312373 1915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:11:25.312412 1915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:11:25.321245 1915 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:37917
I20260926 20:11:25.321266 2043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:37917 every 8 connection(s)
I20260926 20:11:25.322256 1915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:11:25.325759 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 1915
I20260926 20:11:25.325929 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:11:25.326251 23754 external_mini_cluster.cc:1428] Setting key baaf462127f36a3d70a7d625adee4443
Sep 26 20:11:25 dist-test-slave-7034 krb5kdc[1540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453485, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:11:25.336354 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.324038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:38183 (local address 127.23.50.190:44157)
0926 20:11:25.324207 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:25.324210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:25.324991 (+ 781us) server_negotiation.cc:407] Connection header received
0926 20:11:25.326851 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:25.326853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:25.326904 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:25.327008 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:25.328492 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.329072 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.330302 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.330517 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.330598 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.332844 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:25.332847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:25.332849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:25.332872 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:25.334330 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:25.334946 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:25.334949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:25.334950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:25.335027 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:25.335354 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:25.335356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:25.335357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:25.335531 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:25.335609 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:25.336128 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 20:11:25.336232 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:11:25.337230 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.324406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:44157 (local address 127.23.50.131:38183)
0926 20:11:25.324838 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:25.324880 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:25.326604 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:25.327224 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:25.327236 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:25.327784 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:11:25.328340 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.328361 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.329210 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.329213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.330193 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.330203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.330640 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.330643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.330778 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.331306 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:11:25.331322 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:11:25.332696 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:11:25.334447 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:25.334453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:25.334466 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:25.334851 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:25.335124 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:11:25.335127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:11:25.335128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:11:25.335272 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:11:25.335663 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:11:25.335668 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:11:25.335912 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 20:11:25.336083 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 20:11:25.336355 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260926 20:11:25.338510 2044 heartbeater.cc:347] Connected to a master server at 127.23.50.190:44157
I20260926 20:11:25.338734 2044 heartbeater.cc:464] Registering TS with master...
I20260926 20:11:25.339231 2044 heartbeater.cc:511] Master 127.23.50.190:44157 requested a full tablet report, sending...
I20260926 20:11:25.340476 1583 ts_manager.cc:194] Registered new tserver with Master: 64ac651ad7ad4b7c8e6ea73ec94d9a2b (127.23.50.131:37917)
I20260926 20:11:25.341109 1583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:38183
I20260926 20:11:25.352541 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:11:25.361723 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.354578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.23.50.190:44157)
0926 20:11:25.354736 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:25.354739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:25.354828 (+ 89us) server_negotiation.cc:407] Connection header received
0926 20:11:25.354914 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:25.354916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:25.354956 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:25.355024 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:11:25.355741 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.356243 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.357128 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.357350 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.357425 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.358782 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:11:25.358784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:11:25.358786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:11:25.358809 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:11:25.360285 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:25.360665 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:25.360667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:25.360668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:25.360740 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:11:25.360960 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:11:25.360962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:11:25.360963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:11:25.361162 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:11:25.361271 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:25.361438 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 20:11:25.361527 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:11:25.364991 1583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:11:25.368116 1583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:11:25.384037 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.377883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34328 (local address 127.23.50.131:37917)
0926 20:11:25.378227 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:25.378259 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:25.379030 (+ 771us) server_negotiation.cc:407] Connection header received
0926 20:11:25.379112 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:25.379119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:25.379316 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:25.379521 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:25.379881 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.380944 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.382413 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.383260 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.383372 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.383511 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:25.383757 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 20:11:25.383873 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":135,"threads_started":1}
I20260926 20:11:25.384227 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.377205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.23.50.129:45807)
0926 20:11:25.378969 (+ 1764us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:25.378986 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:25.379009 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:11:25.379088 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:25.379096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:25.379312 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:25.379463 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:25.379738 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.381066 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.382564 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.383117 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.383265 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.383436 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:25.383958 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 20:11:25.384062 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1646,"thread_start_us":154,"threads_started":1}
I20260926 20:11:25.384294 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.377889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.23.50.130:39589)
0926 20:11:25.378363 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:25.378374 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:25.378651 (+ 277us) server_negotiation.cc:407] Connection header received
0926 20:11:25.378815 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:25.378820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:25.379054 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:25.379241 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:25.379581 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.380452 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.382505 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.383267 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.383373 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.383511 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:25.383981 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 20:11:25.384090 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":138,"threads_started":1}
I20260926 20:11:25.383883 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.377033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:45807 (local address 127.0.0.1:42686)
0926 20:11:25.378467 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:25.378525 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:25.378669 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:25.379454 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:25.379457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:25.379509 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:25.379626 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.379633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.381200 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.381203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.382414 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.382420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.383271 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.383274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.383403 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.383443 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 20:11:25.383558 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:11:25.383694 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"mutex_wait_us":28,"spinlock_wait_cycles":7552,"thread_start_us":98,"threads_started":1}
I20260926 20:11:25.384785 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.377043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:39589 (local address 127.0.0.1:43202)
0926 20:11:25.378471 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:25.378525 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:25.378669 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:25.379243 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:25.379249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:25.379270 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:25.379444 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.379452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.380587 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.380593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.382391 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.382402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.383577 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.383580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.383661 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.383708 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 20:11:25.383831 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 20:11:25.383903 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1386,"mutex_wait_us":79,"spinlock_wait_cycles":4736,"thread_start_us":90,"threads_started":1}
I20260926 20:11:25.383883 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.377353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:37917 (local address 127.0.0.1:34328)
0926 20:11:25.378906 (+ 1553us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:25.378918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:25.379013 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:25.379657 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:25.379660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:25.379679 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:25.379757 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.379762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.381100 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.381103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.382284 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.382294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.383517 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.383524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.383609 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.383632 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:11:25.383702 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:11:25.383755 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1493,"mutex_wait_us":40,"thread_start_us":91,"threads_started":1}
I20260926 20:11:25.387499 1710 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6e1ce7d97b43888c0326b302d410b8 (DEFAULT_TABLE table=test-table [id=c0a99c0e8e30405887f612e0759085b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:25.387624 1844 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6e1ce7d97b43888c0326b302d410b8 (DEFAULT_TABLE table=test-table [id=c0a99c0e8e30405887f612e0759085b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:25.387631 1978 tablet_service.cc:1467] Processing CreateTablet for tablet 2d6e1ce7d97b43888c0326b302d410b8 (DEFAULT_TABLE table=test-table [id=c0a99c0e8e30405887f612e0759085b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:11:25.388557 1710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6e1ce7d97b43888c0326b302d410b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:25.388616 1844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6e1ce7d97b43888c0326b302d410b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:25.388638 1978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d6e1ce7d97b43888c0326b302d410b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:11:25.395149 2061 tablet_bootstrap.cc:492] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8: Bootstrap starting.
I20260926 20:11:25.396277 2063 tablet_bootstrap.cc:492] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75: Bootstrap starting.
I20260926 20:11:25.397022 2062 tablet_bootstrap.cc:492] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b: Bootstrap starting.
I20260926 20:11:25.397571 2061 tablet_bootstrap.cc:654] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:25.398483 2061 log.cc:824] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:11:25.399026 2063 tablet_bootstrap.cc:654] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:25.399426 2062 tablet_bootstrap.cc:654] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b: Neither blocks nor log segments found. Creating new log.
I20260926 20:11:25.400040 2063 log.cc:824] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75: Log is configured to *not* fsync() on all Append() calls
I20260926 20:11:25.400664 2062 log.cc:824] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:11:25.400717 2061 tablet_bootstrap.cc:492] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8: No bootstrap required, opened a new log
I20260926 20:11:25.400904 2061 ts_tablet_manager.cc:1397] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:11:25.402140 2063 tablet_bootstrap.cc:492] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75: No bootstrap required, opened a new log
I20260926 20:11:25.402338 2063 ts_tablet_manager.cc:1397] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:11:25.402531 2062 tablet_bootstrap.cc:492] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b: No bootstrap required, opened a new log
I20260926 20:11:25.402709 2062 ts_tablet_manager.cc:1397] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:11:25.404469 2061 raft_consensus.cc:348] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5922a655cd14b9086a07e3d4a2e26e8" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45807 } } peers { permanent_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39589 } } peers { permanent_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37917 } }
I20260926 20:11:25.404704 2061 raft_consensus.cc:374] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:25.404786 2061 raft_consensus.cc:729] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5922a655cd14b9086a07e3d4a2e26e8, State: Initialized, Role: FOLLOWER
I20260926 20:11:25.405238 2061 consensus_queue.cc:260] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5922a655cd14b9086a07e3d4a2e26e8" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45807 } } peers { permanent_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39589 } } peers { permanent_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37917 } }
I20260926 20:11:25.406191 2061 ts_tablet_manager.cc:1428] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260926 20:11:25.406262 1776 heartbeater.cc:503] Master 127.23.50.190:44157 was elected leader, sending a full tablet report...
I20260926 20:11:25.406212 2063 raft_consensus.cc:348] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5922a655cd14b9086a07e3d4a2e26e8" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45807 } } peers { permanent_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39589 } } peers { permanent_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37917 } }
I20260926 20:11:25.406534 2063 raft_consensus.cc:374] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:25.406613 2063 raft_consensus.cc:729] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf3e363a0314b61bcd2a0d6493ddc75, State: Initialized, Role: FOLLOWER
I20260926 20:11:25.407043 2063 consensus_queue.cc:260] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5922a655cd14b9086a07e3d4a2e26e8" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45807 } } peers { permanent_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39589 } } peers { permanent_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37917 } }
I20260926 20:11:25.407672 1910 heartbeater.cc:503] Master 127.23.50.190:44157 was elected leader, sending a full tablet report...
I20260926 20:11:25.407541 2062 raft_consensus.cc:348] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5922a655cd14b9086a07e3d4a2e26e8" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45807 } } peers { permanent_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39589 } } peers { permanent_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37917 } }
I20260926 20:11:25.407783 2062 raft_consensus.cc:374] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:11:25.407838 2062 raft_consensus.cc:729] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ac651ad7ad4b7c8e6ea73ec94d9a2b, State: Initialized, Role: FOLLOWER
I20260926 20:11:25.407977 2063 ts_tablet_manager.cc:1428] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:11:25.408236 2062 consensus_queue.cc:260] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5922a655cd14b9086a07e3d4a2e26e8" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45807 } } peers { permanent_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39589 } } peers { permanent_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37917 } }
I20260926 20:11:25.409153 2044 heartbeater.cc:503] Master 127.23.50.190:44157 was elected leader, sending a full tablet report...
I20260926 20:11:25.409153 2062 ts_tablet_manager.cc:1428] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:11:25.438796 2069 raft_consensus.cc:482] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:11:25.439013 2069 raft_consensus.cc:504] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5922a655cd14b9086a07e3d4a2e26e8" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45807 } } peers { permanent_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39589 } } peers { permanent_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37917 } }
I20260926 20:11:25.440210 2069 leader_election.cc:288] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5922a655cd14b9086a07e3d4a2e26e8 (127.23.50.129:45807), 1bf3e363a0314b61bcd2a0d6493ddc75 (127.23.50.130:39589)
I20260926 20:11:25.444864 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.440515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:44301 (local address 127.23.50.129:45807)
0926 20:11:25.440653 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:25.440658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:25.440673 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:11:25.440728 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:25.440734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:25.440793 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:25.440858 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:25.441195 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.442041 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.443552 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.444163 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.444283 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.444335 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:25.444694 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 20:11:25.444761 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:11:25.444865 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.440665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:38653 (local address 127.23.50.130:39589)
0926 20:11:25.440772 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:25.440777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:25.441035 (+ 258us) server_negotiation.cc:407] Connection header received
0926 20:11:25.441116 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:25.441118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:25.441230 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:25.441310 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:11:25.441536 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.442239 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.443552 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.444150 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.444283 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.444335 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:25.444664 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 20:11:25.444738 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 20:11:25.444895 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.440373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:45807 (local address 127.23.50.131:44301)
0926 20:11:25.440497 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:25.440517 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:25.440607 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:25.440903 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:25.440906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:25.440924 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:25.441072 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.441080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.442147 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.442150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.443414 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.443424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.444292 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.444294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.444485 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.444556 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 20:11:25.444631 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:11:25.444745 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":94}
I20260926 20:11:25.444909 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.440484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:39589 (local address 127.23.50.131:38653)
0926 20:11:25.440931 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:25.440943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:11:25.441037 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:11:25.441327 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:11:25.441329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:11:25.441344 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:11:25.441431 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.441435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.442333 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.442336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.443434 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:11:25.443440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.444287 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.444290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:11:25.444480 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.444551 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 20:11:25.444631 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 20:11:25.444745 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":87,"thread_start_us":117,"threads_started":1}
I20260926 20:11:25.445640 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6e1ce7d97b43888c0326b302d410b8" candidate_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5922a655cd14b9086a07e3d4a2e26e8" is_pre_election: true
I20260926 20:11:25.445647 1864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6e1ce7d97b43888c0326b302d410b8" candidate_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75" is_pre_election: true
I20260926 20:11:25.445999 1864 raft_consensus.cc:2454] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64ac651ad7ad4b7c8e6ea73ec94d9a2b in term 0.
I20260926 20:11:25.445999 1730 raft_consensus.cc:2454] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64ac651ad7ad4b7c8e6ea73ec94d9a2b in term 0.
I20260926 20:11:25.446543 1931 leader_election.cc:302] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1bf3e363a0314b61bcd2a0d6493ddc75, 64ac651ad7ad4b7c8e6ea73ec94d9a2b, b5922a655cd14b9086a07e3d4a2e26e8; no voters:
I20260926 20:11:25.446862 2069 raft_consensus.cc:2781] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:11:25.446944 2069 raft_consensus.cc:482] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:11:25.446998 2069 raft_consensus.cc:3037] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:25.448035 2069 raft_consensus.cc:504] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5922a655cd14b9086a07e3d4a2e26e8" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45807 } } peers { permanent_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39589 } } peers { permanent_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37917 } }
I20260926 20:11:25.448452 2069 leader_election.cc:288] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [CANDIDATE]: Term 1 election: Requested vote from peers b5922a655cd14b9086a07e3d4a2e26e8 (127.23.50.129:45807), 1bf3e363a0314b61bcd2a0d6493ddc75 (127.23.50.130:39589)
I20260926 20:11:25.448788 1730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6e1ce7d97b43888c0326b302d410b8" candidate_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5922a655cd14b9086a07e3d4a2e26e8"
I20260926 20:11:25.448849 1864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d6e1ce7d97b43888c0326b302d410b8" candidate_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75"
I20260926 20:11:25.448930 1730 raft_consensus.cc:3037] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:25.448958 1864 raft_consensus.cc:3037] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:11:25.450312 1864 raft_consensus.cc:2454] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ac651ad7ad4b7c8e6ea73ec94d9a2b in term 1.
I20260926 20:11:25.450314 1730 raft_consensus.cc:2454] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64ac651ad7ad4b7c8e6ea73ec94d9a2b in term 1.
I20260926 20:11:25.450632 1932 leader_election.cc:302] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1bf3e363a0314b61bcd2a0d6493ddc75, 64ac651ad7ad4b7c8e6ea73ec94d9a2b, b5922a655cd14b9086a07e3d4a2e26e8; no voters:
I20260926 20:11:25.450829 2069 raft_consensus.cc:2781] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:11:25.451069 2069 raft_consensus.cc:686] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 1 LEADER]: Becoming Leader. State: Replica: 64ac651ad7ad4b7c8e6ea73ec94d9a2b, State: Running, Role: LEADER
I20260926 20:11:25.451416 2069 consensus_queue.cc:237] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5922a655cd14b9086a07e3d4a2e26e8" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45807 } } peers { permanent_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39589 } } peers { permanent_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37917 } }
I20260926 20:11:25.454866 1585 catalog_manager.cc:5495] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 64ac651ad7ad4b7c8e6ea73ec94d9a2b (127.23.50.131). New cstate: current_term: 1 leader_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5922a655cd14b9086a07e3d4a2e26e8" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64ac651ad7ad4b7c8e6ea73ec94d9a2b" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37917 } health_report { overall_health: HEALTHY } } }
I20260926 20:11:25.482746 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.478865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.23.50.131:37917)
0926 20:11:25.479008 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:11:25.479011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:11:25.479025 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:11:25.479151 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:11:25.479153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:11:25.479212 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:11:25.479308 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:11:25.479552 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.480055 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.480970 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:11:25.481172 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:11:25.481288 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:11:25.481645 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:11:25.482192 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:11:25.482294 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:11:25.482460 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 20:11:25.482551 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:11:25.488750 1730 raft_consensus.cc:1261] T 2d6e1ce7d97b43888c0326b302d410b8 P b5922a655cd14b9086a07e3d4a2e26e8 [term 1 FOLLOWER]: Refusing update from remote peer 64ac651ad7ad4b7c8e6ea73ec94d9a2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:11:25.488750 1864 raft_consensus.cc:1261] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75 [term 1 FOLLOWER]: Refusing update from remote peer 64ac651ad7ad4b7c8e6ea73ec94d9a2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:11:25.489568 2069 consensus_queue.cc:1035] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5922a655cd14b9086a07e3d4a2e26e8" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:11:25.489809 2071 consensus_queue.cc:1035] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf3e363a0314b61bcd2a0d6493ddc75" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:11:25.498257 2075 mvcc.cc:205] Tried to move back new op lower bound from 7333697476554461184 to 7333697476412026880. Current Snapshot: MvccSnapshot[applied={T|T < 7333697476554461184}]
I20260926 20:11:25.508466 1583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:11:25.508618 1583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:11:25.511133 1583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32af490af34c4ba58b4e1702653d7ef1: Sending DeleteTablet for 3 replicas of tablet 2d6e1ce7d97b43888c0326b302d410b8
I20260926 20:11:25.511961 1844 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6e1ce7d97b43888c0326b302d410b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:25 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:43202
I20260926 20:11:25.511960 1710 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6e1ce7d97b43888c0326b302d410b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:25 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:42686
I20260926 20:11:25.512120 1978 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d6e1ce7d97b43888c0326b302d410b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:11:25 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:34328
I20260926 20:11:25.512463 2083 tablet_replica.cc:330] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75: stopping tablet replica
I20260926 20:11:25.512661 2085 tablet_replica.cc:330] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b: stopping tablet replica
I20260926 20:11:25.512807 2083 raft_consensus.cc:2229] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:11:25.513006 2085 raft_consensus.cc:2229] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:11:25.513142 2083 raft_consensus.cc:2258] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:25.513424 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 1647
I20260926 20:11:25.513509 2085 raft_consensus.cc:2258] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:11:25.514974 2083 ts_tablet_manager.cc:1905] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:25.516594 2085 ts_tablet_manager.cc:1905] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:11:25.517774 2083 ts_tablet_manager.cc:1918] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:11:25.517916 2083 log.cc:1196] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/2d6e1ce7d97b43888c0326b302d410b8
I20260926 20:11:25.518237 2083 ts_tablet_manager.cc:1939] T 2d6e1ce7d97b43888c0326b302d410b8 P 1bf3e363a0314b61bcd2a0d6493ddc75: Deleting consensus metadata
I20260926 20:11:25.519313 2085 ts_tablet_manager.cc:1918] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:11:25.519470 2085 log.cc:1196] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/2d6e1ce7d97b43888c0326b302d410b8
I20260926 20:11:25.519860 2085 ts_tablet_manager.cc:1939] T 2d6e1ce7d97b43888c0326b302d410b8 P 64ac651ad7ad4b7c8e6ea73ec94d9a2b: Deleting consensus metadata
I20260926 20:11:25.520020 1570 catalog_manager.cc:4841] TS 1bf3e363a0314b61bcd2a0d6493ddc75 (127.23.50.130:39589): tablet 2d6e1ce7d97b43888c0326b302d410b8 (table test-table [id=c0a99c0e8e30405887f612e0759085b1]) successfully deleted
I20260926 20:11:25.521023 1571 catalog_manager.cc:4841] TS 64ac651ad7ad4b7c8e6ea73ec94d9a2b (127.23.50.131:37917): tablet 2d6e1ce7d97b43888c0326b302d410b8 (table test-table [id=c0a99c0e8e30405887f612e0759085b1]) successfully deleted
W20260926 20:11:25.523239 1569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:45807 (error 108)
I20260926 20:11:25.523442 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 1781
I20260926 20:11:25.525161 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:11:25.524813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:45807 (local address 127.0.0.1:42698)
0926 20:11:25.524966 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 20:11:25.525055 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 20:11:25.525521 1569 catalog_manager.cc:4584] TS b5922a655cd14b9086a07e3d4a2e26e8 (127.23.50.129:45807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d6e1ce7d97b43888c0326b302d410b8: Network error: Client connection negotiation failed: client connection to 127.23.50.129:45807: connect: Connection refused (error 111)
I20260926 20:11:25.530534 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 1915
I20260926 20:11:25.536051 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 1553
2026-09-26T20:11:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2312 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:11:25 dist-test-slave-7034 krb5kdc[2089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:11:25 dist-test-slave-7034 krb5kdc[2089](info): set up 2 sockets
Sep 26 20:11:25 dist-test-slave-7034 krb5kdc[2089](info): commencing operation
krb5kdc: starting...
W20260926 20:11:26.579564 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:11:26 dist-test-slave-7034 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:11:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.50.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.50.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/rangeradmin_127.23.50.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/rangeradmin_127.23.50.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.50.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.50.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/rangerlookup_127.23.50.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/rangerlookup_127.23.50.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.50.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.50.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/HTTP_127.23.50.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/HTTP_127.23.50.148@KRBTEST.COM.keytab.
I20260926 20:11:26.762444 23754 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFqzGjO/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/postgres -l logfile start
2026-09-26 20:11:29.360 UTC [2119] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:11:29.360 UTC [2119] LOG: listening on IPv4 address "127.23.50.148", port 60609
2026-09-26 20:11:29.364 UTC [2119] LOG: listening on Unix socket "/tmp/.s.PGSQL.60609"
2026-09-26 20:11:29.386 UTC [2122] LOG: database system was shut down at 2026-09-26 20:11:27 UTC
2026-09-26 20:11:29.390 UTC [2119] LOG: database system is ready to accept connections
I20260926 20:11:30.339300 23754 mini_postgres.cc:96] Postgres bound to 60609
2026-09-26 20:11:30.346 UTC [2130] FATAL: database "slave" does not exist
127.23.50.148:60609 - accepting connections
I20260926 20:11:30.346805 23754 mini_ranger.cc:161] Starting Ranger...
I20260926 20:11:30.362203 23754 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:11:30.885802 23754 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:11:30.885980 23754 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-admin
2026-09-26 20:11:31,194 [I] DB FLAVOR :POSTGRES
2026-09-26 20:11:31,195 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:11:31,195 [I] Checking connection..
2026-09-26 20:11:31,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:11:31,448 [I] Checking connection passed.
2026-09-26 20:11:31,448 [I] --------- Verifying version history table ---------
2026-09-26 20:11:31,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:11:31,696 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:11:31,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:11:31,940 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:11:31,940 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:11:31,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:11:32.191 UTC [2194] WARNING: there is no transaction in progress
2026-09-26 20:11:32,198 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:11:32,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:11:32,449 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:11:32,450 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:11:32,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:11:32,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:11:32,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7034.c.gcp-upstream.internal','N') ;"
2026-09-26 20:11:33,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:11:33,417 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:11:33,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:11:33,674 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:11:33,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:11:33,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:11:34,175 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:11:34,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:11:35.194 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.212 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.230 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.246 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.263 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.283 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.343 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.350 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.365 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.376 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.387 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.402 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.412 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.418 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.425 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:35.432 UTC [2302] WARNING: there is no transaction in progress
2026-09-26 20:11:36,114 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:11:36,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:11:36,372 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:11:36,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:11:36,625 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:11:36,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:11:36,871 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:11:36,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:11:37,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:11:37,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7034.c.gcp-upstream.internal';"
2026-09-26 20:11:37,639 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:11:37,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:11:37,878 [I] DB_PATCHES have already been applied
I20260926 20:11:37.884537 23754 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFqzGjO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:11:37.884632 23754 mini_ranger.cc:193] Using host: 127.23.50.148
I20260926 20:11:37.888134 23754 mini_ranger.cc:227] Ranger admin URL: http://127.23.50.148:43083
Sep 26, 2026 8:11:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:11:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:11:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:11:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:11:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:11:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:11:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:11:39.004242 23754 mini_ranger.cc:160] Time spent starting Ranger: real 8.657s user 0.002s sys 0.010s
Sep 26, 2026 8:11:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:11:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 8:11:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:11:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:11:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:12:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:12:01.207504 23754 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.50.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.50.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/rangerkms_127.23.50.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/rangerkms_127.23.50.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.50.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.50.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.50.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/HTTP_127.23.50.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.50.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/HTTP_127.23.50.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:12:01 dist-test-slave-7034 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453521, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:12:01.292021 23754 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:12:01.301569 23754 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:12:02.237386 23754 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:12:02.237496 23754 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-kms
I20260926 20:12:02.237605 23754 mini_ranger_kms.cc:227] Using postgres at 127.23.50.148:60609
2026-09-26 20:12:02,574 [I] DB FLAVOR :POSTGRES
2026-09-26 20:12:02,574 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:12:02,574 [I] Checking connection
2026-09-26 20:12:02,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:12:02,833 [I] connection success
2026-09-26 20:12:02,833 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:12:02,833 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:12:02,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:12:03,080 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:12:03,080 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:12:03,081 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:12:03,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:60609/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:12:03,362 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:12:03.880069 23754 mini_ranger_kms.cc:310] Created kms service
I20260926 20:12:04.077754 23754 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:12:04.120838 23754 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:12:04 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453521, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:04.520659 23754 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:12:04.520816 23754 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:12:04.520838 23754 mini_ranger_kms.cc:240] Using host: 127.23.50.148
I20260926 20:12:04.524601 23754 mini_ranger_kms.cc:276] Ranger KMS PID: 2482
I20260926 20:12:04.524748 23754 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.50.148:49139
Sep 26, 2026 8:12:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:12:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:12:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:12:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:12:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:12:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 20:12:06 dist-test-slave-7034 krb5kdc[2089](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790453526, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.50.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:12:06.810004 23754 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.518s user 0.018s sys 0.192s
I20260926 20:12:06.810148 23754 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:12:06.810271 23754 mini_ranger_kms.cc:401] 127.23.50.148:49139/kms/v1/keys
Sep 26 20:12:07 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453526, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.50.148@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
Sep 26 20:12:07 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453526, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.50.148@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
Sep 26 20:12:09 dist-test-slave-7034 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:12:09.848517 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:35775
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:35775
--ranger_config_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.50.148:49139/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:12:09.976176 2533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:12:09.976468 2533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:12:09.976507 2533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:12:09.980873 2533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:12:09.980950 2533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:12:09.980970 2533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:12:09.980988 2533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:12:09.981003 2533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:12:09.987195 2533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.50.148:49139/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:35775
--ranger_config_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:35775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:12:09.988426 2533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:12:09.989607 2533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:12:09.998694 2538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:09.998728 2541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:09.998751 2539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:12:09.999353 2533 server_base.cc:1034] running on GCE node
I20260926 20:12:10.000000 2533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:12:10.001227 2533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:12:10.002444 2533 hybrid_clock.cc:648] HybridClock initialized: now 1790453530002418 us; error 52 us; skew 500 ppm
Sep 26 20:12:10 dist-test-slave-7034 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453530, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:12:10.011757 2533 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:12:10.012893 2533 webserver.cc:460] Webserver started at http://127.23.50.190:39033/ using document root <none> and password file <none>
I20260926 20:12:10.013542 2533 fs_manager.cc:359] Metadata directory not provided
I20260926 20:12:10.013603 2533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:12:10.013931 2533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:12:10 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453530, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:10.070380 2533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "bb49d1e5d7664e31957b7789b46a4eff"
format_stamp: "Formatted at 2026-09-26 20:12:10 on dist-test-slave-7034"
server_key: "2245f4d02425f720bd585b5994b6e8e0"
server_key_iv: "e89408365c79f71be85e3588d1b20f07"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:10.071115 2533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "bb49d1e5d7664e31957b7789b46a4eff"
format_stamp: "Formatted at 2026-09-26 20:12:10 on dist-test-slave-7034"
server_key: "2245f4d02425f720bd585b5994b6e8e0"
server_key_iv: "e89408365c79f71be85e3588d1b20f07"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:10.076257 2533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 20:12:10.094290 2548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:10.095959 2533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:12:10.096109 2533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "bb49d1e5d7664e31957b7789b46a4eff"
format_stamp: "Formatted at 2026-09-26 20:12:10 on dist-test-slave-7034"
server_key: "2245f4d02425f720bd585b5994b6e8e0"
server_key_iv: "e89408365c79f71be85e3588d1b20f07"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:10.096272 2533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:12:10.127175 2533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:12:10.131016 2533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:12:10.131239 2533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:12:10.140362 2533 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:35775
I20260926 20:12:10.140483 2600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:35775 every 8 connection(s)
I20260926 20:12:10.141637 2533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:12:10.144778 2601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:12:10.147241 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 2533
I20260926 20:12:10.147385 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:12:10.147686 23754 external_mini_cluster.cc:1428] Setting key 086fdefa0e0fdd0a97727173be9cc2ca
I20260926 20:12:10.151064 2601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff: Bootstrap starting.
I20260926 20:12:10.153702 2601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff: Neither blocks nor log segments found. Creating new log.
I20260926 20:12:10.154721 2601 log.cc:824] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff: Log is configured to *not* fsync() on all Append() calls
I20260926 20:12:10.156744 2601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff: No bootstrap required, opened a new log
Sep 26 20:12:10 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:12:10.160772 2601 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb49d1e5d7664e31957b7789b46a4eff" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35775 } }
I20260926 20:12:10.161034 2601 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:12:10.161118 2601 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb49d1e5d7664e31957b7789b46a4eff, State: Initialized, Role: FOLLOWER
I20260926 20:12:10.161621 2601 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb49d1e5d7664e31957b7789b46a4eff" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35775 } }
I20260926 20:12:10.161775 2601 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:12:10.161878 2601 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:12:10.161978 2601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:12:10.162887 2601 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb49d1e5d7664e31957b7789b46a4eff" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35775 } }
I20260926 20:12:10.163295 2601 leader_election.cc:302] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb49d1e5d7664e31957b7789b46a4eff; no voters:
I20260926 20:12:10.163640 2601 leader_election.cc:288] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:12:10.163784 2606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:12:10.163995 2606 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [term 1 LEADER]: Becoming Leader. State: Replica: bb49d1e5d7664e31957b7789b46a4eff, State: Running, Role: LEADER
I20260926 20:12:10.164058 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:10.149176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.23.50.190:35775)
0926 20:12:10.149730 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:10.149750 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:10.149793 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:12:10.150671 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:10.150698 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:10.151156 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:10.151544 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:12:10.153014 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:10.154576 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:10.155653 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:10.156143 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:10.156330 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:10.158910 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:12:10.158916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:12:10.158931 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:12:10.158957 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:12:10.160617 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:10.161260 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:10.161269 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:10.161276 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:10.161393 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:10.161686 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:10.161692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:10.161697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:10.162330 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:12:10.162560 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:10.162793 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 20:12:10.163085 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":132,"threads_started":1}
I20260926 20:12:10.164840 2601 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:12:10.165210 2606 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb49d1e5d7664e31957b7789b46a4eff" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35775 } }
I20260926 20:12:10.166994 2608 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb49d1e5d7664e31957b7789b46a4eff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb49d1e5d7664e31957b7789b46a4eff" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35775 } } }
I20260926 20:12:10.167038 2607 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb49d1e5d7664e31957b7789b46a4eff. Latest consensus state: current_term: 1 leader_uuid: "bb49d1e5d7664e31957b7789b46a4eff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb49d1e5d7664e31957b7789b46a4eff" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 35775 } } }
I20260926 20:12:10.167176 2608 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [sys.catalog]: This master's current role is: LEADER
I20260926 20:12:10.167274 2607 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff [sys.catalog]: This master's current role is: LEADER
I20260926 20:12:10.167508 2601 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:12:10.167634 2620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:12:10.173453 2620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:12:10.179805 2620 catalog_manager.cc:1347] Generated new cluster ID: b439f72ac110414eb329860d20aefa0f
I20260926 20:12:10.179872 2620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:12:10.204942 2620 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:12:10.206488 2620 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:12:10.229133 2620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff: Generated new TSK 0
I20260926 20:12:10.230192 2620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:12:10.909548 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.50.148:49139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:35775
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:12:11.052057 2646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:12:11.052498 2646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:12:11.052596 2646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:12:11.059969 2646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:12:11.060109 2646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:12:11.060250 2646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:12:11.067337 2646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.50.148:49139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:35775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:12:11.068482 2646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:12:11.069535 2646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:12:11.081048 2656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:11.081362 2653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:11.081123 2654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:12:11.082165 2646 server_base.cc:1034] running on GCE node
I20260926 20:12:11.082666 2646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:12:11.083555 2646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:12:11.084758 2646 hybrid_clock.cc:648] HybridClock initialized: now 1790453531084721 us; error 50 us; skew 500 ppm
Sep 26 20:12:11 dist-test-slave-7034 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:12:11.087661 2646 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:12:11.088532 2646 webserver.cc:460] Webserver started at http://127.23.50.129:35721/ using document root <none> and password file <none>
I20260926 20:12:11.089238 2646 fs_manager.cc:359] Metadata directory not provided
I20260926 20:12:11.089332 2646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:12:11.089624 2646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:12:11 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:11.109197 2646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "7d27a5af5f264a9d960c7aad00c7a60f"
format_stamp: "Formatted at 2026-09-26 20:12:11 on dist-test-slave-7034"
server_key: "821eeca7a148c9974bed3b64f890a559"
server_key_iv: "15370d2660b5bc8b4a95c19e500dced8"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:11.110742 2646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "7d27a5af5f264a9d960c7aad00c7a60f"
format_stamp: "Formatted at 2026-09-26 20:12:11 on dist-test-slave-7034"
server_key: "821eeca7a148c9974bed3b64f890a559"
server_key_iv: "15370d2660b5bc8b4a95c19e500dced8"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:11.115289 2646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:12:11.135824 2663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:11.137511 2646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:12:11.137639 2646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "7d27a5af5f264a9d960c7aad00c7a60f"
format_stamp: "Formatted at 2026-09-26 20:12:11 on dist-test-slave-7034"
server_key: "821eeca7a148c9974bed3b64f890a559"
server_key_iv: "15370d2660b5bc8b4a95c19e500dced8"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:11.137791 2646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:12:11.153077 2646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:12:11.156232 2646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:12:11.156438 2646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:12:11.157058 2646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:12:11.158092 2646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:12:11.158159 2646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:11.158252 2646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:12:11.158308 2646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:11.169324 2646 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:46099
I20260926 20:12:11.169341 2776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:46099 every 8 connection(s)
I20260926 20:12:11.170781 2646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:12:11.177983 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 2646
I20260926 20:12:11.178125 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:12:11.178458 23754 external_mini_cluster.cc:1428] Setting key a834c68d8b62e3bd61c7114ed2ba8f73
Sep 26 20:12:11 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:12:11.186753 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.172901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:33809 (local address 127.23.50.190:35775)
0926 20:12:11.173236 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:11.173246 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:11.173967 (+ 721us) server_negotiation.cc:407] Connection header received
0926 20:12:11.174945 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:11.174950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:11.175012 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:11.175125 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:12:11.176264 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.177143 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.178654 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.179011 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.179129 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.182202 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:12:11.182215 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:12:11.182222 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:12:11.182259 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:12:11.184184 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:11.184764 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:11.184771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:11.184775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:11.184883 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:11.185236 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:11.185242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:11.185247 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:11.185527 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:12:11.185646 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:11.186346 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 20:12:11.186507 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
I20260926 20:12:11.187429 2780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.173296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:35775 (local address 127.23.50.129:33809)
0926 20:12:11.173767 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:11.173823 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:12:11.174710 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:12:11.175323 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:12:11.175331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:12:11.175598 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:12:11.176098 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:11.176113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.177290 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.177293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:11.178502 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:11.178515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.179193 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.179198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:11.179404 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.180144 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:12:11.180169 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:12:11.182036 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:12:11.184323 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:11.184328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:11.184342 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:11.184638 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:11.184978 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:11.184980 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:11.184981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:11.185123 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:11.185680 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:12:11.185686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:12:11.186048 (+ 362us) client_negotiation.cc:769] Sending connection context
0926 20:12:11.186236 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 20:12:11.186535 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":130,"threads_started":1}
I20260926 20:12:11.188937 2777 heartbeater.cc:347] Connected to a master server at 127.23.50.190:35775
I20260926 20:12:11.189265 2777 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:12:11.189908 2777 heartbeater.cc:511] Master 127.23.50.190:35775 requested a full tablet report, sending...
I20260926 20:12:11.191918 2558 ts_manager.cc:194] Registered new tserver with Master: 7d27a5af5f264a9d960c7aad00c7a60f (127.23.50.129:46099)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
I20260926 20:12:11.193636 2558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:33809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:12:11.213725 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.50.148:49139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:35775
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:12:11 dist-test-slave-7034 krb5kdc[2089](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790453531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 20:12:11.378078 2785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:12:11.378504 2785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:12:11.378605 2785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:12:11.385864 2785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:12:11.385972 2785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:12:11.386070 2785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:12:11.394827 2785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.50.148:49139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:35775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:12:11.396514 2785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:12:11.397874 2785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:12:11.410967 2792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:11.411541 2791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:11.412397 2794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:12:11.412747 2785 server_base.cc:1034] running on GCE node
I20260926 20:12:11.413399 2785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:12:11.414294 2785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:12:11.415627 2785 hybrid_clock.cc:648] HybridClock initialized: now 1790453531415598 us; error 39 us; skew 500 ppm
Sep 26 20:12:11 dist-test-slave-7034 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:12:11.419569 2785 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:12:11.420783 2785 webserver.cc:460] Webserver started at http://127.23.50.130:39655/ using document root <none> and password file <none>
I20260926 20:12:11.421584 2785 fs_manager.cc:359] Metadata directory not provided
I20260926 20:12:11.421720 2785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:12:11.422055 2785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:12:11 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:11.446036 2785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "09242d94eb974320b8e1d43cf67bc2a3"
format_stamp: "Formatted at 2026-09-26 20:12:11 on dist-test-slave-7034"
server_key: "635340064c5c44b5dd9be67895fc004c"
server_key_iv: "c0b3b3fd48e872e2ffc18bd04bca07b7"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:11.447031 2785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "09242d94eb974320b8e1d43cf67bc2a3"
format_stamp: "Formatted at 2026-09-26 20:12:11 on dist-test-slave-7034"
server_key: "635340064c5c44b5dd9be67895fc004c"
server_key_iv: "c0b3b3fd48e872e2ffc18bd04bca07b7"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:11.475976 2785 fs_manager.cc:691] Time spent creating directory manager: real 0.028s user 0.002s sys 0.002s
I20260926 20:12:11.497359 2801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:11.499141 2785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:12:11.499285 2785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "09242d94eb974320b8e1d43cf67bc2a3"
format_stamp: "Formatted at 2026-09-26 20:12:11 on dist-test-slave-7034"
server_key: "635340064c5c44b5dd9be67895fc004c"
server_key_iv: "c0b3b3fd48e872e2ffc18bd04bca07b7"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:11.499410 2785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:12:11.519310 2785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:12:11.522861 2785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:12:11.523094 2785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:12:11.523761 2785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:12:11.524806 2785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:12:11.524853 2785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:11.524917 2785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:12:11.524950 2785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:11.536244 2785 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:38425
I20260926 20:12:11.536322 2914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:38425 every 8 connection(s)
I20260926 20:12:11.537312 2785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:12:11.547514 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 2785
I20260926 20:12:11.547652 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:12:11.547969 23754 external_mini_cluster.cc:1428] Setting key 49796a2c66766e9ff7b1cc52bfd62a66
Sep 26 20:12:11 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:12:11.564167 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.539318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:33093 (local address 127.23.50.190:35775)
0926 20:12:11.539488 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:11.539493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:11.540438 (+ 945us) server_negotiation.cc:407] Connection header received
0926 20:12:11.541403 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:11.541407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:11.541472 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:11.541590 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:12:11.542809 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.543722 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.555346 (+ 11624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.555773 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.555896 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.559227 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:12:11.559232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:12:11.559236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:12:11.559270 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:12:11.561232 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:11.561983 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:11.561988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:11.561991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:11.562087 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:11.562596 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:11.562601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:11.562603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:11.562890 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:12:11.562986 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:11.563760 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 20:12:11.563922 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 20:12:11.565843 2917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.539760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:35775 (local address 127.23.50.130:33093)
0926 20:12:11.540253 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:11.540309 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:12:11.541171 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:12:11.541776 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:12:11.541783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:12:11.542129 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:12:11.542650 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:11.542666 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.553680 (+ 11014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.553686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:11.555163 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:11.555179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.556109 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.556116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:11.556361 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.557086 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:12:11.557113 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:12:11.558988 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:12:11.561396 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:11.561405 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:11.561429 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:11.561833 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:11.562211 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:11.562214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:11.562216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:11.562415 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:11.563062 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:12:11.563070 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:12:11.563451 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 20:12:11.564423 (+ 972us) client_negotiation.cc:240] Negotiation successful
0926 20:12:11.564796 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":139,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
I20260926 20:12:11.567131 2915 heartbeater.cc:347] Connected to a master server at 127.23.50.190:35775
I20260926 20:12:11.567375 2915 heartbeater.cc:464] Registering TS with master...
I20260926 20:12:11.567991 2915 heartbeater.cc:511] Master 127.23.50.190:35775 requested a full tablet report, sending...
I20260926 20:12:11.569131 2558 ts_manager.cc:194] Registered new tserver with Master: 09242d94eb974320b8e1d43cf67bc2a3 (127.23.50.130:38425)
I20260926 20:12:11.570001 2558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:33093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:12:11.593581 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.50.148:49139/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:35775
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:12:11 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
W20260926 20:12:11.759653 2920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:12:11.760017 2920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:12:11.760078 2920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:12:11.765583 2920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:12:11.765692 2920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:12:11.765803 2920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:12:11.771703 2920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.50.148:49139/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:35775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:12:11.772857 2920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:12:11.774005 2920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:12:11.783905 2925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:11.784006 2926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:11.784190 2928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:12:11.785235 2920 server_base.cc:1034] running on GCE node
I20260926 20:12:11.785758 2920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:12:11.786595 2920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:12:11.787822 2920 hybrid_clock.cc:648] HybridClock initialized: now 1790453531787785 us; error 48 us; skew 500 ppm
Sep 26 20:12:11 dist-test-slave-7034 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:12:11.790900 2920 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:12:11.791998 2920 webserver.cc:460] Webserver started at http://127.23.50.131:39595/ using document root <none> and password file <none>
I20260926 20:12:11.792573 2920 fs_manager.cc:359] Metadata directory not provided
I20260926 20:12:11.792672 2920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:12:11.792886 2920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:12:11 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:11.809618 2920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "dd45a01507674cf184e4cb8e839a11e6"
format_stamp: "Formatted at 2026-09-26 20:12:11 on dist-test-slave-7034"
server_key: "de978457cc35ebc3918b74257506f2ac"
server_key_iv: "2ecc469f89a059add14afb377b1a53a3"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:11.810401 2920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "dd45a01507674cf184e4cb8e839a11e6"
format_stamp: "Formatted at 2026-09-26 20:12:11 on dist-test-slave-7034"
server_key: "de978457cc35ebc3918b74257506f2ac"
server_key_iv: "2ecc469f89a059add14afb377b1a53a3"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:11.815280 2920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 20:12:11.834546 2935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:11.836053 2920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:12:11.836207 2920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "dd45a01507674cf184e4cb8e839a11e6"
format_stamp: "Formatted at 2026-09-26 20:12:11 on dist-test-slave-7034"
server_key: "de978457cc35ebc3918b74257506f2ac"
server_key_iv: "2ecc469f89a059add14afb377b1a53a3"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:11.836364 2920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:12:11.859701 2920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:12:11.863641 2920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:12:11.863863 2920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:12:11.864675 2920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:12:11.865728 2920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:12:11.865790 2920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:11.865880 2920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:12:11.865914 2920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:11.879489 2920 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:37439
I20260926 20:12:11.880749 2920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:12:11.886138 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 2920
I20260926 20:12:11.886262 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:12:11.886559 23754 external_mini_cluster.cc:1428] Setting key f4bdae7de61fc1e9bba15e0f5f2cd886
I20260926 20:12:11.882156 3048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:37439 every 8 connection(s)
Sep 26 20:12:11 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:12:11.908876 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.886976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:60727 (local address 127.23.50.190:35775)
0926 20:12:11.887381 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:11.887387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:11.894496 (+ 7109us) server_negotiation.cc:407] Connection header received
0926 20:12:11.895870 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:11.895874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:11.895935 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:11.895991 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:12:11.897955 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.898790 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.900566 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.900924 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.901030 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.904355 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:12:11.904360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:12:11.904363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:12:11.904393 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:12:11.906199 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:11.906891 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:11.906894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:11.906896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:11.906989 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:11.907444 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:11.907447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:11.907449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:11.907693 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:12:11.907777 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:11.908546 (+ 769us) server_negotiation.cc:299] Negotiation successful
0926 20:12:11.908695 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312}
I20260926 20:12:11.911149 3051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.884789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:35775 (local address 127.23.50.131:60727)
0926 20:12:11.894259 (+ 9470us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:11.894345 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0926 20:12:11.895570 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:12:11.896478 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:12:11.896488 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:12:11.896916 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:12:11.897711 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:11.897738 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.898965 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.898968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:11.900405 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:11.900417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.901262 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.901267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:11.901490 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.902290 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:12:11.902318 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:12:11.904141 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:12:11.906348 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:11.906355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:11.906374 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:11.906760 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:11.907115 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:11.907118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:11.907120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:11.907316 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:11.907846 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:12:11.907852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:12:11.908232 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 20:12:11.909552 (+ 1320us) client_negotiation.cc:240] Negotiation successful
0926 20:12:11.909977 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9155,"thread_start_us":144,"threads_started":1}
I20260926 20:12:11.912976 3049 heartbeater.cc:347] Connected to a master server at 127.23.50.190:35775
I20260926 20:12:11.913239 3049 heartbeater.cc:464] Registering TS with master...
I20260926 20:12:11.914328 3049 heartbeater.cc:511] Master 127.23.50.190:35775 requested a full tablet report, sending...
I20260926 20:12:11.915537 2558 ts_manager.cc:194] Registered new tserver with Master: dd45a01507674cf184e4cb8e839a11e6 (127.23.50.131:37439)
I20260926 20:12:11.916298 2558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:60727
I20260926 20:12:11.926347 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:12:11.943975 2779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.929053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.23.50.190:35775)
0926 20:12:11.929176 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:11.929180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:11.934119 (+ 4939us) server_negotiation.cc:407] Connection header received
0926 20:12:11.934344 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:11.934347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:11.934406 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:11.934465 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:12:11.935507 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.936305 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.937570 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.937963 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.938043 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.939987 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:12:11.939992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:12:11.939995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:12:11.940025 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:12:11.941825 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:11.942451 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:11.942455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:11.942457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:11.942559 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:11.942915 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:11.942920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:11.942922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:11.943163 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:12:11.943276 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:11.943681 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 20:12:11.943804 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:12:11.948231 2558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:12:11.953068 2558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:12:11.973412 3063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.965285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:46099 (local address 127.0.0.1:49906)
0926 20:12:11.967394 (+ 2109us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:11.967452 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:12:11.967584 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:12:11.968356 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:12:11.968362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:12:11.968400 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:12:11.968568 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:11.968576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.969956 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.969959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:11.971425 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:11.971437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.972770 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.972774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:11.972937 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.972983 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:12:11.973097 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:12:11.973229 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1913,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260926 20:12:11.974071 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.965408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.23.50.129:46099)
0926 20:12:11.967891 (+ 2483us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:11.967897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:11.967918 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:12:11.967982 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:11.967986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:11.968172 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:11.968396 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:12:11.968713 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.969805 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.971568 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.972431 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.972534 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.972707 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:11.973799 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0926 20:12:11.973930 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2360,"thread_start_us":54,"threads_started":1}
I20260926 20:12:11.979264 3065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.967160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:38425 (local address 127.0.0.1:40282)
0926 20:12:11.973019 (+ 5859us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:11.973034 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:12:11.973114 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:12:11.974055 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:12:11.974058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:12:11.974079 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:12:11.974200 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:11.974205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.975593 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.975596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:11.977542 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:11.977554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.978864 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.978868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:11.978958 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.978983 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:12:11.979056 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 20:12:11.979128 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5795,"thread_start_us":93,"threads_started":1}
I20260926 20:12:11.978869 2711 tablet_service.cc:1467] Processing CreateTablet for tablet 0919585aea4848d7bec88e3377c66ded (DEFAULT_TABLE table=test-table [id=ccc4b0ee73034a9884651780f3ec17c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:12:11.979724 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.967462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40282 (local address 127.23.50.130:38425)
0926 20:12:11.973370 (+ 5908us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:11.973382 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:11.973405 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:12:11.973483 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:11.973490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:11.973703 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:11.973903 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:12:11.974329 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.975465 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.977708 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.978534 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.978641 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.978783 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:11.979442 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 20:12:11.979549 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5799,"thread_start_us":138,"threads_started":1}
I20260926 20:12:11.982780 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.966328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.23.50.131:37439)
0926 20:12:11.967114 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:11.967131 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:11.968744 (+ 1613us) server_negotiation.cc:407] Connection header received
0926 20:12:11.969739 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:11.969744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:11.970002 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:11.970133 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:12:11.970541 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.971626 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.980289 (+ 8663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.981967 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.982054 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.982188 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:11.982539 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 20:12:11.982640 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":149,"threads_started":1}
I20260926 20:12:11.983263 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:11.965908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:37439 (local address 127.0.0.1:55454)
0926 20:12:11.968650 (+ 2742us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:11.968665 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:12:11.969595 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:12:11.970303 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:12:11.970306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:12:11.970323 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:12:11.970413 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:11.970419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.971753 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.971757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:11.975398 (+ 3641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:11.975409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:11.982287 (+ 6878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:11.982290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:11.982381 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:11.982402 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:12:11.983037 (+ 635us) client_negotiation.cc:240] Negotiation successful
0926 20:12:11.983119 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2662,"mutex_wait_us":1596,"thread_start_us":73,"threads_started":1}
I20260926 20:12:11.983144 2849 tablet_service.cc:1467] Processing CreateTablet for tablet 0919585aea4848d7bec88e3377c66ded (DEFAULT_TABLE table=test-table [id=ccc4b0ee73034a9884651780f3ec17c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:12:11.984313 2849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0919585aea4848d7bec88e3377c66ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:12:11.986364 2982 tablet_service.cc:1467] Processing CreateTablet for tablet 0919585aea4848d7bec88e3377c66ded (DEFAULT_TABLE table=test-table [id=ccc4b0ee73034a9884651780f3ec17c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:12:11.987437 2982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0919585aea4848d7bec88e3377c66ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:12:11.991494 2711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0919585aea4848d7bec88e3377c66ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:12:12.001672 3067 tablet_bootstrap.cc:492] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3: Bootstrap starting.
I20260926 20:12:12.003332 3068 tablet_bootstrap.cc:492] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f: Bootstrap starting.
I20260926 20:12:12.004925 3067 tablet_bootstrap.cc:654] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3: Neither blocks nor log segments found. Creating new log.
I20260926 20:12:12.006008 3067 log.cc:824] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:12:12.008047 3068 tablet_bootstrap.cc:654] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f: Neither blocks nor log segments found. Creating new log.
I20260926 20:12:12.008562 3067 tablet_bootstrap.cc:492] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3: No bootstrap required, opened a new log
I20260926 20:12:12.008857 3067 ts_tablet_manager.cc:1397] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260926 20:12:12.008987 3068 log.cc:824] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:12:12.010116 3069 tablet_bootstrap.cc:492] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6: Bootstrap starting.
I20260926 20:12:12.013017 3069 tablet_bootstrap.cc:654] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6: Neither blocks nor log segments found. Creating new log.
I20260926 20:12:12.013233 3067 raft_consensus.cc:348] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 46099 } } peers { permanent_uuid: "dd45a01507674cf184e4cb8e839a11e6" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37439 } } peers { permanent_uuid: "09242d94eb974320b8e1d43cf67bc2a3" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38425 } }
I20260926 20:12:12.013624 3067 raft_consensus.cc:374] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:12:12.013744 3067 raft_consensus.cc:729] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09242d94eb974320b8e1d43cf67bc2a3, State: Initialized, Role: FOLLOWER
I20260926 20:12:12.014045 3069 log.cc:824] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:12:12.014500 3067 consensus_queue.cc:260] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 46099 } } peers { permanent_uuid: "dd45a01507674cf184e4cb8e839a11e6" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37439 } } peers { permanent_uuid: "09242d94eb974320b8e1d43cf67bc2a3" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38425 } }
I20260926 20:12:12.015741 3067 ts_tablet_manager.cc:1428] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260926 20:12:12.016148 2915 heartbeater.cc:503] Master 127.23.50.190:35775 was elected leader, sending a full tablet report...
I20260926 20:12:12.016341 3068 tablet_bootstrap.cc:492] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f: No bootstrap required, opened a new log
I20260926 20:12:12.016543 3068 ts_tablet_manager.cc:1397] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20260926 20:12:12.021025 3068 raft_consensus.cc:348] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 46099 } } peers { permanent_uuid: "dd45a01507674cf184e4cb8e839a11e6" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37439 } } peers { permanent_uuid: "09242d94eb974320b8e1d43cf67bc2a3" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38425 } }
I20260926 20:12:12.021332 3068 raft_consensus.cc:374] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:12:12.021399 3068 raft_consensus.cc:729] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d27a5af5f264a9d960c7aad00c7a60f, State: Initialized, Role: FOLLOWER
I20260926 20:12:12.022300 3068 consensus_queue.cc:260] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 46099 } } peers { permanent_uuid: "dd45a01507674cf184e4cb8e839a11e6" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37439 } } peers { permanent_uuid: "09242d94eb974320b8e1d43cf67bc2a3" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38425 } }
I20260926 20:12:12.023309 3069 tablet_bootstrap.cc:492] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6: No bootstrap required, opened a new log
I20260926 20:12:12.023399 3068 ts_tablet_manager.cc:1428] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:12:12.023604 3069 ts_tablet_manager.cc:1397] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260926 20:12:12.023757 2777 heartbeater.cc:503] Master 127.23.50.190:35775 was elected leader, sending a full tablet report...
I20260926 20:12:12.028983 3069 raft_consensus.cc:348] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 46099 } } peers { permanent_uuid: "dd45a01507674cf184e4cb8e839a11e6" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37439 } } peers { permanent_uuid: "09242d94eb974320b8e1d43cf67bc2a3" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38425 } }
I20260926 20:12:12.029313 3069 raft_consensus.cc:374] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:12:12.029409 3069 raft_consensus.cc:729] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd45a01507674cf184e4cb8e839a11e6, State: Initialized, Role: FOLLOWER
I20260926 20:12:12.029918 3069 consensus_queue.cc:260] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 46099 } } peers { permanent_uuid: "dd45a01507674cf184e4cb8e839a11e6" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37439 } } peers { permanent_uuid: "09242d94eb974320b8e1d43cf67bc2a3" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38425 } }
I20260926 20:12:12.031052 3069 ts_tablet_manager.cc:1428] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260926 20:12:12.031452 3049 heartbeater.cc:503] Master 127.23.50.190:35775 was elected leader, sending a full tablet report...
W20260926 20:12:12.038699 2916 tablet.cc:2369] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 20:12:12 dist-test-slave-7034 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:12.079010 3074 raft_consensus.cc:482] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:12:12.079308 3074 raft_consensus.cc:504] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 46099 } } peers { permanent_uuid: "dd45a01507674cf184e4cb8e839a11e6" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37439 } } peers { permanent_uuid: "09242d94eb974320b8e1d43cf67bc2a3" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38425 } }
I20260926 20:12:12.080816 3074 leader_election.cc:288] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd45a01507674cf184e4cb8e839a11e6 (127.23.50.131:37439), 09242d94eb974320b8e1d43cf67bc2a3 (127.23.50.130:38425)
I20260926 20:12:12.086865 3064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:12.081888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:36271 (local address 127.23.50.131:37439)
0926 20:12:12.082019 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:12.082025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:12.082243 (+ 218us) server_negotiation.cc:407] Connection header received
0926 20:12:12.082400 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:12.082404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:12.082456 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:12.082534 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:12:12.082969 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:12.083784 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:12.085463 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:12.086120 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:12.086224 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:12.086277 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:12.086642 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 20:12:12.086723 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:12:12.087525 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:12.081048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:37439 (local address 127.23.50.129:36271)
0926 20:12:12.082127 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:12.082143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:12:12.082283 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:12:12.082655 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:12:12.082658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:12:12.082692 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:12:12.082833 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:12.082841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:12.083954 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:12.083957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:12.085319 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:12.085331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:12.086385 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:12.086388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:12.086488 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:12.086511 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:12:12.087281 (+ 770us) client_negotiation.cc:240] Negotiation successful
0926 20:12:12.087365 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"thread_start_us":147,"threads_started":1}
I20260926 20:12:12.088353 2987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0919585aea4848d7bec88e3377c66ded" candidate_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd45a01507674cf184e4cb8e839a11e6" is_pre_election: true
I20260926 20:12:12.088690 2987 raft_consensus.cc:2454] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d27a5af5f264a9d960c7aad00c7a60f in term 0.
I20260926 20:12:12.089306 2666 leader_election.cc:302] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d27a5af5f264a9d960c7aad00c7a60f, dd45a01507674cf184e4cb8e839a11e6; no voters:
I20260926 20:12:12.089625 3074 raft_consensus.cc:2781] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:12:12.089706 3074 raft_consensus.cc:482] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:12:12.089752 3074 raft_consensus.cc:3037] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:12:12.091140 3074 raft_consensus.cc:504] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 46099 } } peers { permanent_uuid: "dd45a01507674cf184e4cb8e839a11e6" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37439 } } peers { permanent_uuid: "09242d94eb974320b8e1d43cf67bc2a3" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38425 } }
I20260926 20:12:12.091514 3074 leader_election.cc:288] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [CANDIDATE]: Term 1 election: Requested vote from peers dd45a01507674cf184e4cb8e839a11e6 (127.23.50.131:37439), 09242d94eb974320b8e1d43cf67bc2a3 (127.23.50.130:38425)
I20260926 20:12:12.092239 2987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0919585aea4848d7bec88e3377c66ded" candidate_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd45a01507674cf184e4cb8e839a11e6"
I20260926 20:12:12.092370 2987 raft_consensus.cc:3037] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:12:12.093093 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:12.081546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:38425 (local address 127.23.50.129:43235)
0926 20:12:12.086531 (+ 4985us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:12.086547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:12:12.086737 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:12:12.087160 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:12:12.087163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:12:12.087184 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:12:12.087294 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:12.087300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:12.090777 (+ 3477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:12.090781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:12.091540 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:12.091549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:12.092493 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:12.092497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:12.092602 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:12.092627 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:12:12.092851 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 20:12:12.092938 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4905,"thread_start_us":98,"threads_started":1}
I20260926 20:12:12.093537 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:12.081667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:43235 (local address 127.23.50.130:38425)
0926 20:12:12.081797 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:12.081802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:12.086636 (+ 4834us) server_negotiation.cc:407] Connection header received
0926 20:12:12.086880 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:12.086886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:12.086956 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:12.087007 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:12:12.089905 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:12.090620 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:12.091690 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:12.092257 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:12.092378 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:12.092430 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:12.093326 (+ 896us) server_negotiation.cc:299] Negotiation successful
0926 20:12:12.093416 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:12:12.093830 2987 raft_consensus.cc:2454] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d27a5af5f264a9d960c7aad00c7a60f in term 1.
I20260926 20:12:12.094286 2869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0919585aea4848d7bec88e3377c66ded" candidate_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09242d94eb974320b8e1d43cf67bc2a3" is_pre_election: true
I20260926 20:12:12.094600 2869 raft_consensus.cc:2454] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d27a5af5f264a9d960c7aad00c7a60f in term 0.
I20260926 20:12:12.094961 2868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0919585aea4848d7bec88e3377c66ded" candidate_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09242d94eb974320b8e1d43cf67bc2a3"
I20260926 20:12:12.095059 2868 raft_consensus.cc:3037] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:12:12.095633 2666 leader_election.cc:302] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d27a5af5f264a9d960c7aad00c7a60f, dd45a01507674cf184e4cb8e839a11e6; no voters:
I20260926 20:12:12.095878 3074 raft_consensus.cc:2781] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:12:12.096104 3074 raft_consensus.cc:686] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [term 1 LEADER]: Becoming Leader. State: Replica: 7d27a5af5f264a9d960c7aad00c7a60f, State: Running, Role: LEADER
I20260926 20:12:12.096349 2868 raft_consensus.cc:2454] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d27a5af5f264a9d960c7aad00c7a60f in term 1.
I20260926 20:12:12.096472 3074 consensus_queue.cc:237] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 46099 } } peers { permanent_uuid: "dd45a01507674cf184e4cb8e839a11e6" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37439 } } peers { permanent_uuid: "09242d94eb974320b8e1d43cf67bc2a3" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38425 } }
I20260926 20:12:12.100158 2559 catalog_manager.cc:5495] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d27a5af5f264a9d960c7aad00c7a60f (127.23.50.129). New cstate: current_term: 1 leader_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d27a5af5f264a9d960c7aad00c7a60f" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd45a01507674cf184e4cb8e839a11e6" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09242d94eb974320b8e1d43cf67bc2a3" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38425 } health_report { overall_health: UNKNOWN } } }
W20260926 20:12:12.132880 3050 tablet.cc:2369] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:12:12.137447 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:12.133103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.23.50.129:46099)
0926 20:12:12.133235 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:12.133240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:12.133254 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:12:12.133342 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:12.133345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:12.133417 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:12.133527 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:12:12.133898 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:12.134738 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:12.135803 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:12.136046 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:12.136163 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:12.136496 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:12:12.136948 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:12:12.137091 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:12.137222 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 20:12:12.137296 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:12:12.162477 2868 raft_consensus.cc:1261] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3 [term 1 FOLLOWER]: Refusing update from remote peer 7d27a5af5f264a9d960c7aad00c7a60f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:12:12.163379 3080 consensus_queue.cc:1035] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "09242d94eb974320b8e1d43cf67bc2a3" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:12:12.167032 2987 raft_consensus.cc:1261] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6 [term 1 FOLLOWER]: Refusing update from remote peer 7d27a5af5f264a9d960c7aad00c7a60f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:12:12.167735 3080 consensus_queue.cc:1035] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd45a01507674cf184e4cb8e839a11e6" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:12:12.172812 2778 tablet.cc:2369] T 0919585aea4848d7bec88e3377c66ded P 7d27a5af5f264a9d960c7aad00c7a60f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:12:12.188091 3083 mvcc.cc:205] Tried to move back new op lower bound from 7333697667728072704 to 7333697667470221312. Current Snapshot: MvccSnapshot[applied={T|T < 7333697667728072704}]
I20260926 20:12:12.190897 3086 mvcc.cc:205] Tried to move back new op lower bound from 7333697667728072704 to 7333697667470221312. Current Snapshot: MvccSnapshot[applied={T|T < 7333697667728072704}]
I20260926 20:12:12.195292 3084 mvcc.cc:205] Tried to move back new op lower bound from 7333697667728072704 to 7333697667470221312. Current Snapshot: MvccSnapshot[applied={T|T < 7333697667728072704}]
I20260926 20:12:12.208153 2559 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:12:12.208374 2559 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:12:12.211412 2559 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb49d1e5d7664e31957b7789b46a4eff: Sending DeleteTablet for 3 replicas of tablet 0919585aea4848d7bec88e3377c66ded
I20260926 20:12:12.212277 2711 tablet_service.cc:1514] Processing DeleteTablet for tablet 0919585aea4848d7bec88e3377c66ded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:12:12 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:49906
I20260926 20:12:12.213352 2982 tablet_service.cc:1514] Processing DeleteTablet for tablet 0919585aea4848d7bec88e3377c66ded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:12:12 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:55454
I20260926 20:12:12.214356 2849 tablet_service.cc:1514] Processing DeleteTablet for tablet 0919585aea4848d7bec88e3377c66ded with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:12:12 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:40282
I20260926 20:12:12.215854 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 2646
I20260926 20:12:12.221323 3094 tablet_replica.cc:330] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3: stopping tablet replica
I20260926 20:12:12.221746 3094 raft_consensus.cc:2229] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:12:12.222172 3094 raft_consensus.cc:2258] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:12:12.222621 3093 tablet_replica.cc:330] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6: stopping tablet replica
I20260926 20:12:12.222945 3093 raft_consensus.cc:2229] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:12:12.223270 3093 raft_consensus.cc:2258] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:12:12.231125 3094 ts_tablet_manager.cc:1905] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:12:12.233815 3093 ts_tablet_manager.cc:1905] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:12:12.236943 3093 ts_tablet_manager.cc:1918] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:12:12.237073 3093 log.cc:1196] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/0919585aea4848d7bec88e3377c66ded
I20260926 20:12:12.237432 3093 ts_tablet_manager.cc:1939] T 0919585aea4848d7bec88e3377c66ded P dd45a01507674cf184e4cb8e839a11e6: Deleting consensus metadata
I20260926 20:12:12.237833 3094 ts_tablet_manager.cc:1918] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:12:12.239092 2551 catalog_manager.cc:4841] TS dd45a01507674cf184e4cb8e839a11e6 (127.23.50.131:37439): tablet 0919585aea4848d7bec88e3377c66ded (table test-table [id=ccc4b0ee73034a9884651780f3ec17c7]) successfully deleted
W20260926 20:12:12.240144 2549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:46099 (error 108)
I20260926 20:12:12.240360 3094 log.cc:1196] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/0919585aea4848d7bec88e3377c66ded
I20260926 20:12:12.240847 3094 ts_tablet_manager.cc:1939] T 0919585aea4848d7bec88e3377c66ded P 09242d94eb974320b8e1d43cf67bc2a3: Deleting consensus metadata
I20260926 20:12:12.241560 3061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:12.241277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:46099 (local address 127.0.0.1:49920)
0926 20:12:12.241395 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:12.241474 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 20:12:12.241762 2549 catalog_manager.cc:4584] TS 7d27a5af5f264a9d960c7aad00c7a60f (127.23.50.129:46099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0919585aea4848d7bec88e3377c66ded: Network error: Client connection negotiation failed: client connection to 127.23.50.129:46099: connect: Connection refused (error 111)
I20260926 20:12:12.242167 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 2785
I20260926 20:12:12.249517 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 2920
I20260926 20:12:12.258858 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 2533
I20260926 20:12:12.266379 23754 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:12:12.601459 23754 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:12:12.601585 23754 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:12:12.947062 23754 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:12:12.947 UTC [2119] LOG: received smart shutdown request
2026-09-26 20:12:12.950 UTC [2119] LOG: background worker "logical replication launcher" (PID 2128) exited with exit code 1
2026-09-26 20:12:12.951 UTC [2123] LOG: shutting down
2026-09-26 20:12:13.136 UTC [2119] LOG: database system is shut down
2026-09-26T20:12:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47716 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:12:13 dist-test-slave-7034 krb5kdc[3104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:12:13 dist-test-slave-7034 krb5kdc[3104](info): set up 2 sockets
Sep 26 20:12:13 dist-test-slave-7034 krb5kdc[3104](info): commencing operation
krb5kdc: starting...
W20260926 20:12:14.322824 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:12:14 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:12:14Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.50.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.50.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/rangeradmin_127.23.50.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/rangeradmin_127.23.50.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.50.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.50.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/rangerlookup_127.23.50.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/rangerlookup_127.23.50.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.50.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.50.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/HTTP_127.23.50.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/HTTP_127.23.50.148@KRBTEST.COM.keytab.
I20260926 20:12:14.501541 23754 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFqzGjO/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/postgres -l logfile start
2026-09-26 20:12:17.198 UTC [3134] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 20:12:17.198 UTC [3134] LOG: listening on IPv4 address "127.23.50.148", port 37559
2026-09-26 20:12:17.202 UTC [3134] LOG: listening on Unix socket "/tmp/.s.PGSQL.37559"
2026-09-26 20:12:17.222 UTC [3137] LOG: database system was shut down at 2026-09-26 20:12:15 UTC
2026-09-26 20:12:17.226 UTC [3134] LOG: database system is ready to accept connections
I20260926 20:12:18.190433 23754 mini_postgres.cc:96] Postgres bound to 37559
2026-09-26 20:12:18.197 UTC [3145] FATAL: database "slave" does not exist
127.23.50.148:37559 - accepting connections
I20260926 20:12:18.197726 23754 mini_ranger.cc:161] Starting Ranger...
I20260926 20:12:18.212466 23754 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 20:12:18.734629 23754 mini_ranger.cc:87] Created ranger Postgres database
I20260926 20:12:18.734706 23754 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-admin
2026-09-26 20:12:19,006 [I] DB FLAVOR :POSTGRES
2026-09-26 20:12:19,007 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:12:19,007 [I] Checking connection..
2026-09-26 20:12:19,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 20:12:19,257 [I] Checking connection passed.
2026-09-26 20:12:19,258 [I] --------- Verifying version history table ---------
2026-09-26 20:12:19,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:12:19,500 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:12:19,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:12:19,752 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 20:12:19,753 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 20:12:19,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 20:12:19.996 UTC [3209] WARNING: there is no transaction in progress
2026-09-26 20:12:20,002 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 20:12:20,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 20:12:20,235 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 20:12:20,236 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:12:20,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 20:12:20,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 20:12:20,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7034.c.gcp-upstream.internal','N') ;"
2026-09-26 20:12:20,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:12:21,236 [I] Table x_portal_user does not exist in database ranger
2026-09-26 20:12:21,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:12:21,505 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 20:12:21,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:12:21,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:12:21,972 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 20:12:21,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 20:12:22.967 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:22.986 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.003 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.018 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.034 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.053 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.112 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.119 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.135 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.147 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.159 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.174 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.184 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.193 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.201 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23.209 UTC [3317] WARNING: there is no transaction in progress
2026-09-26 20:12:23,926 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 20:12:23,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 20:12:24,151 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 20:12:24,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 20:12:24,418 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 20:12:24,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 20:12:24,671 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 20:12:24,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 20:12:24,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 20:12:25,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7034.c.gcp-upstream.internal';"
2026-09-26 20:12:25,396 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 20:12:25,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 20:12:25,658 [I] DB_PATCHES have already been applied
I20260926 20:12:25.663805 23754 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFqzGjO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 20:12:25.663900 23754 mini_ranger.cc:193] Using host: 127.23.50.148
I20260926 20:12:25.667598 23754 mini_ranger.cc:227] Ranger admin URL: http://127.23.50.148:52877
Sep 26, 2026 8:12:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:12:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:12:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 8:12:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 8:12:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 8:12:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 8:12:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 20:12:26.768318 23754 mini_ranger.cc:160] Time spent starting Ranger: real 8.571s user 0.007s sys 0.003s
Sep 26, 2026 8:12:46 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 8:12:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 8:12:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 8:12:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 8:12:46 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 8:12:47 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 20:12:48.257555 23754 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.50.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.50.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/rangerkms_127.23.50.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.50.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/rangerkms_127.23.50.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.50.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.50.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.50.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/HTTP_127.23.50.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.50.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/HTTP_127.23.50.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 20:12:48 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453568, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 20:12:48.351230 23754 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 20:12:48.361783 23754 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 20:12:49.400452 23754 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 20:12:49.400542 23754 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-kms
I20260926 20:12:49.400564 23754 mini_ranger_kms.cc:227] Using postgres at 127.23.50.148:37559
2026-09-26 20:12:49,576 [I] DB FLAVOR :POSTGRES
2026-09-26 20:12:49,576 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 20:12:49,576 [I] Checking connection
2026-09-26 20:12:49,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 20:12:49,827 [I] connection success
2026-09-26 20:12:49,828 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 20:12:49,828 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 20:12:49,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 20:12:50,071 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 20:12:50,071 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 20:12:50,071 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 20:12:50,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.50.148:37559/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 20:12:50,341 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 20:12:50.809901 23754 mini_ranger_kms.cc:310] Created kms service
I20260926 20:12:51.018850 23754 mini_ranger_kms.cc:326] Created kudu user
I20260926 20:12:51.066870 23754 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 20:12:51 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453568, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:51.465761 23754 mini_ranger_kms.cc:384] Added ranger policy
I20260926 20:12:51.465947 23754 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFqzGjO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFqzGjO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 20:12:51.465976 23754 mini_ranger_kms.cc:240] Using host: 127.23.50.148
I20260926 20:12:51.469728 23754 mini_ranger_kms.cc:276] Ranger KMS PID: 3497
I20260926 20:12:51.469815 23754 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.50.148:42585
Sep 26, 2026 8:12:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 8:12:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 8:12:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 8:12:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 8:12:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 8:12:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 20:12:53.778298 23754 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.427s user 0.030s sys 0.129s
I20260926 20:12:53.778443 23754 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 20:12:53.778512 23754 mini_ranger_kms.cc:401] 127.23.50.148:42585/kms/v1/keys
Sep 26 20:12:54 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790453574, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.50.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 20:12:54 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453574, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.50.148@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
Sep 26 20:12:54 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453574, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.50.148@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
Sep 26 20:12:57 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:12:57.232918 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:38107
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:38107
--ranger_config_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.50.148:42585/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:12:57.345125 3548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:12:57.345468 3548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:12:57.345556 3548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:12:57.349747 3548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:12:57.349859 3548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:12:57.349915 3548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:12:57.349973 3548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:12:57.350028 3548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:12:57.355983 3548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.50.148:42585/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:38107
--ranger_config_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:12:57.357132 3548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:12:57.358245 3548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:12:57.366683 3554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:57.366690 3553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:57.366714 3556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:12:57.367070 3548 server_base.cc:1034] running on GCE node
I20260926 20:12:57.367599 3548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:12:57.368716 3548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:12:57.369918 3548 hybrid_clock.cc:648] HybridClock initialized: now 1790453577369885 us; error 46 us; skew 500 ppm
Sep 26 20:12:57 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453577, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:12:57.372728 3548 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:12:57.373592 3548 webserver.cc:460] Webserver started at http://127.23.50.190:35619/ using document root <none> and password file <none>
I20260926 20:12:57.374189 3548 fs_manager.cc:359] Metadata directory not provided
I20260926 20:12:57.374267 3548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:12:57.374521 3548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:12:57 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453577, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:57.428086 3548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "eb756fdee4654a909d2de5b2be276726"
format_stamp: "Formatted at 2026-09-26 20:12:57 on dist-test-slave-7034"
server_key: "a1143d68af8242bbec36fe2647df2772"
server_key_iv: "feb78135111c633937f40b9b8bf2fc21"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:57.428851 3548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "eb756fdee4654a909d2de5b2be276726"
format_stamp: "Formatted at 2026-09-26 20:12:57 on dist-test-slave-7034"
server_key: "a1143d68af8242bbec36fe2647df2772"
server_key_iv: "feb78135111c633937f40b9b8bf2fc21"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:57.433878 3548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 20:12:57.451300 3563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:57.452749 3548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:12:57.452910 3548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "eb756fdee4654a909d2de5b2be276726"
format_stamp: "Formatted at 2026-09-26 20:12:57 on dist-test-slave-7034"
server_key: "a1143d68af8242bbec36fe2647df2772"
server_key_iv: "feb78135111c633937f40b9b8bf2fc21"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:57.453055 3548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:12:57.478626 3548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:12:57.481768 3548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:12:57.482034 3548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:12:57.490247 3548 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:38107
I20260926 20:12:57.490253 3615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:38107 every 8 connection(s)
I20260926 20:12:57.491303 3548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:12:57.494165 3616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:12:57.499557 3616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726: Bootstrap starting.
I20260926 20:12:57.501597 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 3548
I20260926 20:12:57.501730 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:12:57.502049 23754 external_mini_cluster.cc:1428] Setting key 8b3e174285a86891c61cd40c6df50d58
I20260926 20:12:57.502223 3616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726: Neither blocks nor log segments found. Creating new log.
I20260926 20:12:57.503867 3616 log.cc:824] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726: Log is configured to *not* fsync() on all Append() calls
I20260926 20:12:57.506299 3616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726: No bootstrap required, opened a new log
I20260926 20:12:57.510269 3616 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb756fdee4654a909d2de5b2be276726" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38107 } }
I20260926 20:12:57.510643 3616 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:12:57.510739 3616 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb756fdee4654a909d2de5b2be276726, State: Initialized, Role: FOLLOWER
I20260926 20:12:57.511265 3616 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb756fdee4654a909d2de5b2be276726" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38107 } }
I20260926 20:12:57.511459 3616 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:12:57.511588 3616 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:12:57.511708 3616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 20:12:57 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:12:57.512744 3616 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb756fdee4654a909d2de5b2be276726" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38107 } }
I20260926 20:12:57.513098 3616 leader_election.cc:302] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb756fdee4654a909d2de5b2be276726; no voters:
I20260926 20:12:57.513454 3616 leader_election.cc:288] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:12:57.513531 3621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:12:57.513824 3621 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 1 LEADER]: Becoming Leader. State: Replica: eb756fdee4654a909d2de5b2be276726, State: Running, Role: LEADER
I20260926 20:12:57.514195 3621 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb756fdee4654a909d2de5b2be276726" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38107 } }
I20260926 20:12:57.514904 3616 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:12:57.515470 3621 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb756fdee4654a909d2de5b2be276726. Latest consensus state: current_term: 1 leader_uuid: "eb756fdee4654a909d2de5b2be276726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb756fdee4654a909d2de5b2be276726" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38107 } } }
I20260926 20:12:57.515588 3621 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [sys.catalog]: This master's current role is: LEADER
I20260926 20:12:57.515793 3621 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb756fdee4654a909d2de5b2be276726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb756fdee4654a909d2de5b2be276726" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38107 } } }
I20260926 20:12:57.515888 3621 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [sys.catalog]: This master's current role is: LEADER
I20260926 20:12:57.517511 3616 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:12:57.520535 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:57.503259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43754 (local address 127.23.50.190:38107)
0926 20:12:57.503679 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:57.503699 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:57.503743 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:12:57.504527 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:57.504546 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:57.504999 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:57.506103 (+ 1104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:12:57.506526 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:57.508026 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:57.509298 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:57.510074 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:57.510230 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:57.513329 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:12:57.513335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:12:57.513354 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:12:57.513386 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:12:57.515489 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:57.516056 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:57.516061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:57.516068 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:57.516198 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:57.516512 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:57.516515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:57.516517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:57.517045 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:12:57.517238 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:57.517573 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 20:12:57.517907 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":136,"threads_started":1}
I20260926 20:12:57.520846 3625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:12:57.526564 3625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:12:57.539793 3625 catalog_manager.cc:1347] Generated new cluster ID: 93acbbd89f02485ea1b892e794e70619
I20260926 20:12:57.539867 3625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:12:57.549806 3625 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:12:57.550858 3625 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:12:57.555652 3625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726: Generated new TSK 0
I20260926 20:12:57.556788 3625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:12:58.301476 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.50.148:42585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:38107
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:12:58.426565 3661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:12:58.426841 3661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:12:58.426918 3661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:12:58.431365 3661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:12:58.431435 3661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:12:58.431499 3661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:12:58.437383 3661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.50.148:42585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:38107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:12:58.438688 3661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:12:58.439731 3661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:12:58.450006 3667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:58.450754 3669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:58.451117 3666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:12:58.451458 3661 server_base.cc:1034] running on GCE node
I20260926 20:12:58.452037 3661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:12:58.452814 3661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:12:58.454005 3661 hybrid_clock.cc:648] HybridClock initialized: now 1790453578453931 us; error 87 us; skew 500 ppm
Sep 26 20:12:58 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453578, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:12:58.457715 3661 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:12:58.458930 3661 webserver.cc:460] Webserver started at http://127.23.50.129:37981/ using document root <none> and password file <none>
I20260926 20:12:58.459612 3661 fs_manager.cc:359] Metadata directory not provided
I20260926 20:12:58.459689 3661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:12:58.459906 3661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:12:58 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453578, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:58.476902 3661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "15e73738ee324d1ebb9717143e0241df"
format_stamp: "Formatted at 2026-09-26 20:12:58 on dist-test-slave-7034"
server_key: "d83b9fa32a14bc968c7329b689d0344b"
server_key_iv: "e5855641a2cafa044125f22ee9c30792"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:58.477667 3661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "15e73738ee324d1ebb9717143e0241df"
format_stamp: "Formatted at 2026-09-26 20:12:58 on dist-test-slave-7034"
server_key: "d83b9fa32a14bc968c7329b689d0344b"
server_key_iv: "e5855641a2cafa044125f22ee9c30792"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:58.482857 3661 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 20:12:58.503726 3678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:58.505406 3661 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260926 20:12:58.505602 3661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "15e73738ee324d1ebb9717143e0241df"
format_stamp: "Formatted at 2026-09-26 20:12:58 on dist-test-slave-7034"
server_key: "d83b9fa32a14bc968c7329b689d0344b"
server_key_iv: "e5855641a2cafa044125f22ee9c30792"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:58.505836 3661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:12:58.540160 3661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:12:58.544008 3661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:12:58.544265 3661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:12:58.545066 3661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:12:58.546203 3661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:12:58.546312 3661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:58.546464 3661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:12:58.546548 3661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:58.557504 3661 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:41537
I20260926 20:12:58.558902 3661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:12:58.560406 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 3661
I20260926 20:12:58.560523 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:12:58.560820 23754 external_mini_cluster.cc:1428] Setting key f211b589003e96bca659039ca3fa1e61
I20260926 20:12:58.561331 3793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:41537 every 8 connection(s)
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
Sep 26 20:12:58 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453578, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
I20260926 20:12:58.584088 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:58.563763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:38107 (local address 127.23.50.129:52339)
0926 20:12:58.566033 (+ 2270us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:58.566093 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:12:58.567152 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:12:58.567922 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:12:58.567932 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:12:58.568312 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:12:58.569001 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:58.569021 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:58.570251 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:58.570254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:58.571720 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:58.571732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:58.574326 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:58.574331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:58.574588 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:58.575527 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:12:58.575550 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:12:58.577453 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:12:58.580199 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:58.580207 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:58.580228 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:58.580548 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:58.580924 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:58.580930 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:58.580935 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:58.581171 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:58.581735 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:12:58.581745 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:12:58.582180 (+ 435us) client_negotiation.cc:769] Sending connection context
0926 20:12:58.582414 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 20:12:58.582818 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2004,"thread_start_us":157,"threads_started":1}
I20260926 20:12:58.584874 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:58.564913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:52339 (local address 127.23.50.190:38107)
0926 20:12:58.565724 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:58.565729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:58.567463 (+ 1734us) server_negotiation.cc:407] Connection header received
0926 20:12:58.567514 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:58.567517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:58.567573 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:58.567677 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:12:58.569185 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:58.570095 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:58.571871 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:58.572233 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:58.572339 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:58.577739 (+ 5400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:12:58.577744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:12:58.577748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:12:58.577783 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:12:58.580026 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:58.580697 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:58.580702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:58.580704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:58.580806 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:58.581299 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:58.581302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:58.581304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:58.581575 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:12:58.581674 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:58.584545 (+ 2871us) server_negotiation.cc:299] Negotiation successful
0926 20:12:58.584684 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":87,"threads_started":1}
I20260926 20:12:58.585925 3794 heartbeater.cc:347] Connected to a master server at 127.23.50.190:38107
I20260926 20:12:58.586181 3794 heartbeater.cc:464] Registering TS with master...
I20260926 20:12:58.586706 3794 heartbeater.cc:511] Master 127.23.50.190:38107 requested a full tablet report, sending...
I20260926 20:12:58.588369 3580 ts_manager.cc:194] Registered new tserver with Master: 15e73738ee324d1ebb9717143e0241df (127.23.50.129:41537)
I20260926 20:12:58.590085 3580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:52339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:12:58.603147 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.50.148:42585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:38107
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:12:58 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790453578, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 20:12:58.732537 3800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:12:58.732856 3800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:12:58.732978 3800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:12:58.738348 3800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:12:58.738457 3800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:12:58.738569 3800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:12:58.744650 3800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.50.148:42585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:38107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:12:58.745968 3800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:12:58.747377 3800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:12:58.758790 3807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:58.758883 3806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:58.758790 3809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:12:58.759984 3800 server_base.cc:1034] running on GCE node
I20260926 20:12:58.760428 3800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:12:58.761144 3800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:12:58.762333 3800 hybrid_clock.cc:648] HybridClock initialized: now 1790453578762298 us; error 47 us; skew 500 ppm
Sep 26 20:12:58 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453578, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:12:58.765401 3800 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:12:58.766453 3800 webserver.cc:460] Webserver started at http://127.23.50.130:36467/ using document root <none> and password file <none>
I20260926 20:12:58.767093 3800 fs_manager.cc:359] Metadata directory not provided
I20260926 20:12:58.767200 3800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:12:58.767452 3800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:12:58 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453578, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:58.789614 3800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "eaf0ed99d2804a33bff3866533eb46c8"
format_stamp: "Formatted at 2026-09-26 20:12:58 on dist-test-slave-7034"
server_key: "90c2cc8d3da031ccc8414f16ebb7efbb"
server_key_iv: "d1e6a1f1c0224a5bc4ff1e673f83b1ef"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:58.790342 3800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "eaf0ed99d2804a33bff3866533eb46c8"
format_stamp: "Formatted at 2026-09-26 20:12:58 on dist-test-slave-7034"
server_key: "90c2cc8d3da031ccc8414f16ebb7efbb"
server_key_iv: "d1e6a1f1c0224a5bc4ff1e673f83b1ef"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:58.795070 3800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:12:58.818269 3816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:58.820277 3800 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260926 20:12:58.820521 3800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "eaf0ed99d2804a33bff3866533eb46c8"
format_stamp: "Formatted at 2026-09-26 20:12:58 on dist-test-slave-7034"
server_key: "90c2cc8d3da031ccc8414f16ebb7efbb"
server_key_iv: "d1e6a1f1c0224a5bc4ff1e673f83b1ef"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:58.820834 3800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:12:58.840466 3800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:12:58.844460 3800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:12:58.844738 3800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:12:58.845577 3800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:12:58.846786 3800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:12:58.846917 3800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:58.847034 3800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:12:58.847127 3800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:58.859505 3800 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:42061
I20260926 20:12:58.860538 3800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:12:58.860875 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 3800
I20260926 20:12:58.861024 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:12:58.861330 23754 external_mini_cluster.cc:1428] Setting key bae8e6a7178a1be6e26b653cc19dc591
I20260926 20:12:58.870484 3929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:42061 every 8 connection(s)
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
Sep 26 20:12:58 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453578, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:12:58.888718 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:58.872846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:44799 (local address 127.23.50.190:38107)
0926 20:12:58.873025 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:58.873030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:58.874123 (+ 1093us) server_negotiation.cc:407] Connection header received
0926 20:12:58.875493 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:58.875500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:58.875574 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:58.875642 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:12:58.877698 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:58.878625 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:58.879995 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:58.880417 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:58.880538 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:58.883963 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:12:58.883969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:12:58.883975 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:12:58.884010 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:12:58.886133 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:58.886812 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:58.886816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:58.886818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:58.886918 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:58.887335 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:58.887341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:58.887344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:58.887538 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:12:58.887674 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:58.888310 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 20:12:58.888539 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 20:12:58.889676 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:58.873297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:38107 (local address 127.23.50.130:44799)
0926 20:12:58.873959 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:58.874017 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 20:12:58.875134 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:12:58.876126 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:12:58.876139 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:12:58.876538 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:12:58.877502 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:58.877522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:58.878800 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:58.878806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:58.879855 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:58.879865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:58.880742 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:58.880748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:58.880998 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:58.881780 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:12:58.881810 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:12:58.883735 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:12:58.886298 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:58.886304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:58.886324 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:58.886675 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:58.887049 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:58.887052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:58.887054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:58.887218 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:58.887675 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:12:58.887683 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:12:58.888021 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 20:12:58.888253 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 20:12:58.888596 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":266,"threads_started":1}
I20260926 20:12:58.891188 3930 heartbeater.cc:347] Connected to a master server at 127.23.50.190:38107
I20260926 20:12:58.891444 3930 heartbeater.cc:464] Registering TS with master...
I20260926 20:12:58.892036 3930 heartbeater.cc:511] Master 127.23.50.190:38107 requested a full tablet report, sending...
I20260926 20:12:58.893239 3580 ts_manager.cc:194] Registered new tserver with Master: eaf0ed99d2804a33bff3866533eb46c8 (127.23.50.130:42061)
I20260926 20:12:58.894054 3580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:44799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:12:58.898276 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.50.148:42585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:38107
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:12:59.038480 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:12:59.038801 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:12:59.038883 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:12:59.043249 3935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:12:59.043342 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:12:59.043448 3935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:12:59.050505 3935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.50.148:42585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:38107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:12:59.051769 3935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:12:59.053244 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:12:59.068977 3941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:59.069011 3943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:59.069412 3940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:12:59.069373 3935 server_base.cc:1034] running on GCE node
I20260926 20:12:59.069974 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:12:59.070676 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:12:59.071897 3935 hybrid_clock.cc:648] HybridClock initialized: now 1790453579071860 us; error 49 us; skew 500 ppm
Sep 26 20:12:59 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453579, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:12:59.075017 3935 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:12:59.076131 3935 webserver.cc:460] Webserver started at http://127.23.50.131:39149/ using document root <none> and password file <none>
I20260926 20:12:59.076730 3935 fs_manager.cc:359] Metadata directory not provided
I20260926 20:12:59.076824 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:12:59.077050 3935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 20:12:59 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453579, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:59.095019 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "48c24133df504b22a8c2c76187a1da05"
format_stamp: "Formatted at 2026-09-26 20:12:59 on dist-test-slave-7034"
server_key: "8d593721b7484b04fe4eea1c3b1a7282"
server_key_iv: "0a893fe57b710ad5505a9c558da87571"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:59.096025 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "48c24133df504b22a8c2c76187a1da05"
format_stamp: "Formatted at 2026-09-26 20:12:59 on dist-test-slave-7034"
server_key: "8d593721b7484b04fe4eea1c3b1a7282"
server_key_iv: "0a893fe57b710ad5505a9c558da87571"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:59.106379 3935 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
Sep 26 20:12:59 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453578, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:59.142292 3950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:59.147024 3935 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.002s sys 0.001s
I20260926 20:12:59.147166 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "48c24133df504b22a8c2c76187a1da05"
format_stamp: "Formatted at 2026-09-26 20:12:59 on dist-test-slave-7034"
server_key: "8d593721b7484b04fe4eea1c3b1a7282"
server_key_iv: "0a893fe57b710ad5505a9c558da87571"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:59.147317 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:12:59.179543 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:12:59.183498 3935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:12:59.183713 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:12:59.184564 3935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:12:59.185775 3935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:12:59.185866 3935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:59.185940 3935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:12:59.185973 3935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:59.197443 3935 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:42035
I20260926 20:12:59.198807 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:12:59.207036 4063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:42035 every 8 connection(s)
I20260926 20:12:59.207422 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 3935
I20260926 20:12:59.207544 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:12:59.207870 23754 external_mini_cluster.cc:1428] Setting key a7731d0b9d62612ed464c036113058a8
Sep 26 20:12:59 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453579, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:12:59.232290 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:59.210868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:33985 (local address 127.23.50.190:38107)
0926 20:12:59.211001 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:59.211005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:59.215204 (+ 4199us) server_negotiation.cc:407] Connection header received
0926 20:12:59.216913 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:59.216933 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:59.216994 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:59.217052 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:12:59.219033 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:59.219879 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:59.223725 (+ 3846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:59.224080 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:59.224162 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:59.227610 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:12:59.227614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:12:59.227619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:12:59.227652 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:12:59.229559 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:59.230325 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:59.230329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:59.230331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:59.230422 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:59.230886 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:59.230890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:59.230892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:59.231132 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:12:59.231217 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:59.231957 (+ 740us) server_negotiation.cc:299] Negotiation successful
0926 20:12:59.232111 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:12:59.236215 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:59.209675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:38107 (local address 127.23.50.131:33985)
0926 20:12:59.214990 (+ 5315us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:59.215077 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0926 20:12:59.216601 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:12:59.217557 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:12:59.217569 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:12:59.218041 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:12:59.218821 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:59.218844 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:59.220049 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:59.220052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:59.223551 (+ 3499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:59.223568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:59.224407 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:59.224413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:59.224634 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:59.225415 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:12:59.225437 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:12:59.227392 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:12:59.229735 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:59.229742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:59.229761 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:59.230198 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:59.230543 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:59.230546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:59.230548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:59.230759 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:59.231286 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:12:59.231293 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:12:59.231633 (+ 340us) client_negotiation.cc:769] Sending connection context
0926 20:12:59.234565 (+ 2932us) client_negotiation.cc:240] Negotiation successful
0926 20:12:59.235036 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4997,"thread_start_us":172,"threads_started":1}
I20260926 20:12:59.237619 4064 heartbeater.cc:347] Connected to a master server at 127.23.50.190:38107
I20260926 20:12:59.237936 4064 heartbeater.cc:464] Registering TS with master...
I20260926 20:12:59.238547 4064 heartbeater.cc:511] Master 127.23.50.190:38107 requested a full tablet report, sending...
I20260926 20:12:59.239807 3580 ts_manager.cc:194] Registered new tserver with Master: 48c24133df504b22a8c2c76187a1da05 (127.23.50.131:42035)
I20260926 20:12:59.240576 3580 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:33985
I20260926 20:12:59.246584 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:12:59.247376 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 3661
I20260926 20:12:59.254199 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 3800
I20260926 20:12:59.260365 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 3935
I20260926 20:12:59.266049 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 3548
I20260926 20:12:59.271888 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:38107
--webserver_interface=127.23.50.190
--webserver_port=35619
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:38107
--ranger_config_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.50.148:42585/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.50.148:42585/kms with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:12:59.382092 4067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:12:59.382411 4067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:12:59.382490 4067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:12:59.386598 4067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:12:59.386675 4067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:12:59.386713 4067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:12:59.386768 4067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:12:59.386806 4067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:12:59.392261 4067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.50.148:42585/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:38107
--ranger_config_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=35619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:12:59.393299 4067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:12:59.394373 4067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:12:59.402763 4073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:12:59.402863 4067 server_base.cc:1034] running on GCE node
W20260926 20:12:59.403014 4072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:59.402760 4075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:12:59.403750 4067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:12:59.404838 4067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:12:59.406039 4067 hybrid_clock.cc:648] HybridClock initialized: now 1790453579406018 us; error 34 us; skew 500 ppm
Sep 26 20:12:59 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453579, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:12:59.408883 4067 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:12:59.409709 4067 webserver.cc:460] Webserver started at http://127.23.50.190:35619/ using document root <none> and password file <none>
I20260926 20:12:59.410269 4067 fs_manager.cc:359] Metadata directory not provided
I20260926 20:12:59.410336 4067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:12:59 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453579, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:59.458695 4067 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:12:59.461031 4083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:59.462615 4067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:12:59.462760 4067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "eb756fdee4654a909d2de5b2be276726"
format_stamp: "Formatted at 2026-09-26 20:12:57 on dist-test-slave-7034"
server_key: "a1143d68af8242bbec36fe2647df2772"
server_key_iv: "feb78135111c633937f40b9b8bf2fc21"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:59.463332 4067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:12:59.478652 4067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:12:59.481711 4067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:12:59.481946 4067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:12:59.489236 4067 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:38107
I20260926 20:12:59.489269 4135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:38107 every 8 connection(s)
I20260926 20:12:59.490334 4067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:12:59.494521 4136 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:12:59.496140 4136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726: Bootstrap starting.
I20260926 20:12:59.499434 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 4067
I20260926 20:12:59.499922 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.50.148:42585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:41537
--local_ip_for_outbound_sockets=127.23.50.129
--tserver_master_addrs=127.23.50.190:38107
--webserver_port=37981
--webserver_interface=127.23.50.129
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.50.148:42585/kms with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:12:59.512861 4136 log.cc:824] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726: Log is configured to *not* fsync() on all Append() calls
I20260926 20:12:59.519572 4136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:12:59.519975 4136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726: Bootstrap complete.
I20260926 20:12:59.524185 4136 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb756fdee4654a909d2de5b2be276726" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38107 } }
I20260926 20:12:59.524612 4136 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb756fdee4654a909d2de5b2be276726, State: Initialized, Role: FOLLOWER
I20260926 20:12:59.525168 4136 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb756fdee4654a909d2de5b2be276726" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38107 } }
I20260926 20:12:59.525321 4136 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:12:59.525408 4136 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:12:59.525524 4136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:12:59.527493 4136 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb756fdee4654a909d2de5b2be276726" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38107 } }
I20260926 20:12:59.527833 4136 leader_election.cc:302] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb756fdee4654a909d2de5b2be276726; no voters:
I20260926 20:12:59.528175 4136 leader_election.cc:288] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:12:59.528366 4140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:12:59.528712 4140 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [term 2 LEADER]: Becoming Leader. State: Replica: eb756fdee4654a909d2de5b2be276726, State: Running, Role: LEADER
I20260926 20:12:59.529072 4140 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb756fdee4654a909d2de5b2be276726" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38107 } }
I20260926 20:12:59.529434 4136 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:12:59.530658 4141 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eb756fdee4654a909d2de5b2be276726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb756fdee4654a909d2de5b2be276726" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38107 } } }
I20260926 20:12:59.530668 4142 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb756fdee4654a909d2de5b2be276726. Latest consensus state: current_term: 2 leader_uuid: "eb756fdee4654a909d2de5b2be276726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb756fdee4654a909d2de5b2be276726" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38107 } } }
I20260926 20:12:59.530815 4142 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [sys.catalog]: This master's current role is: LEADER
I20260926 20:12:59.530817 4141 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726 [sys.catalog]: This master's current role is: LEADER
I20260926 20:12:59.531848 4136 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 20:12:59.532033 4149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:12:59.538467 4149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:12:59.539225 4149 catalog_manager.cc:1259] Loaded cluster ID: 93acbbd89f02485ea1b892e794e70619
I20260926 20:12:59.539367 4149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:12:59.541975 4149 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:12:59.542750 4149 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726: Loaded TSK: 0
I20260926 20:12:59.544152 4149 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:12:59.645071 4138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:12:59.645531 4138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:12:59.645689 4138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:12:59.650682 4138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:12:59.650774 4138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:12:59.650883 4138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:12:59.656581 4138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.50.148:42585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:41537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=37981
--tserver_master_addrs=127.23.50.190:38107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:12:59.657769 4138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:12:59.658995 4138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:12:59.669906 4174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:59.669940 4173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:59.670421 4176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:12:59.670675 4138 server_base.cc:1034] running on GCE node
I20260926 20:12:59.671134 4138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:12:59.671943 4138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:12:59.675216 4138 hybrid_clock.cc:648] HybridClock initialized: now 1790453579675138 us; error 88 us; skew 500 ppm
Sep 26 20:12:59 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453579, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:12:59.678905 4138 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:12:59.680166 4138 webserver.cc:460] Webserver started at http://127.23.50.129:37981/ using document root <none> and password file <none>
I20260926 20:12:59.681013 4138 fs_manager.cc:359] Metadata directory not provided
I20260926 20:12:59.681187 4138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:12:59 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453579, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:12:59.714121 4138 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:12:59.717576 4184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:59.719326 4138 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:12:59.719453 4138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "15e73738ee324d1ebb9717143e0241df"
format_stamp: "Formatted at 2026-09-26 20:12:58 on dist-test-slave-7034"
server_key: "d83b9fa32a14bc968c7329b689d0344b"
server_key_iv: "e5855641a2cafa044125f22ee9c30792"
server_key_version: "kuduclusterkey@0"
I20260926 20:12:59.720213 4138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:12:59.740581 4138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:12:59.744535 4138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:12:59.744850 4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:12:59.745642 4138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:12:59.746783 4138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:12:59.746882 4138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:59.746966 4138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:12:59.747022 4138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:12:59.761145 4138 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:41537
I20260926 20:12:59.762666 4138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:12:59.763365 4297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:41537 every 8 connection(s)
I20260926 20:12:59.772930 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 4138
I20260926 20:12:59.773449 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.50.148:42585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:42061
--local_ip_for_outbound_sockets=127.23.50.130
--tserver_master_addrs=127.23.50.190:38107
--webserver_port=36467
--webserver_interface=127.23.50.130
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.50.148:42585/kms with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:12:59 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453579, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:12:59.786116 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:59.766289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:38107 (local address 127.23.50.129:60511)
0926 20:12:59.767554 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0926 20:12:59.767613 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:12:59.768671 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:12:59.771134 (+ 2463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:12:59.771144 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:12:59.771516 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:12:59.772207 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:59.772230 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:59.774243 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:59.774247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:59.775365 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:12:59.775376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:12:59.776568 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:12:59.776571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:12:59.776715 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:59.777356 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:12:59.777372 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:12:59.778865 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:12:59.781707 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:59.781717 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:59.781736 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:59.782175 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:59.782672 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:12:59.782678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:12:59.782682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:12:59.782881 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:12:59.783782 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:12:59.783791 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:12:59.784160 (+ 369us) client_negotiation.cc:769] Sending connection context
0926 20:12:59.784400 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 20:12:59.784802 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":147,"threads_started":1}
I20260926 20:12:59.789096 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:12:59.766234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:60511 (local address 127.23.50.190:38107)
0926 20:12:59.769412 (+ 3178us) server_negotiation.cc:206] Beginning negotiation
0926 20:12:59.769435 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 20:12:59.769482 (+ 47us) server_negotiation.cc:407] Connection header received
0926 20:12:59.769793 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:12:59.769809 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:12:59.770265 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:12:59.770632 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:12:59.772440 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:59.774075 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:59.775517 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:12:59.776175 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:12:59.776352 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:12:59.779085 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:12:59.779096 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:12:59.779116 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:12:59.779152 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:12:59.781543 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:59.782363 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:59.782373 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:59.782382 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:59.782546 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:12:59.783021 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:12:59.783026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:12:59.783031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:12:59.783624 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:12:59.787333 (+ 3709us) server_negotiation.cc:1035] Waiting for connection context
0926 20:12:59.787537 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:12:59.787908 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3009,"thread_start_us":136,"threads_started":1}
I20260926 20:12:59.792527 4298 heartbeater.cc:347] Connected to a master server at 127.23.50.190:38107
I20260926 20:12:59.792788 4298 heartbeater.cc:464] Registering TS with master...
I20260926 20:12:59.793395 4298 heartbeater.cc:511] Master 127.23.50.190:38107 requested a full tablet report, sending...
W20260926 20:12:59.795919 4298 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:38107 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:12:59.796049 4298 heartbeater.cc:464] Registering TS with master...
I20260926 20:12:59.796248 4298 heartbeater.cc:511] Master 127.23.50.190:38107 requested a full tablet report, sending...
I20260926 20:12:59.797374 4298 heartbeater.cc:464] Registering TS with master...
I20260926 20:12:59.797539 4298 heartbeater.cc:511] Master 127.23.50.190:38107 requested a full tablet report, sending...
W20260926 20:12:59.798369 4298 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:12:59.941931 4302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:12:59.942246 4302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:12:59.942296 4302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:12:59.946980 4302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:12:59.947048 4302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:12:59.947141 4302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:12:59.953083 4302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.50.148:42585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:42061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=36467
--tserver_master_addrs=127.23.50.190:38107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:12:59.954357 4302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:12:59.955431 4302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:12:59.968384 4310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:59.968500 4307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:12:59.968398 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:12:59.969307 4302 server_base.cc:1034] running on GCE node
I20260926 20:12:59.969866 4302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:12:59.970777 4302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:12:59.972030 4302 hybrid_clock.cc:648] HybridClock initialized: now 1790453579971993 us; error 55 us; skew 500 ppm
Sep 26 20:12:59 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453579, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:12:59.976163 4302 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:12:59.977291 4302 webserver.cc:460] Webserver started at http://127.23.50.130:36467/ using document root <none> and password file <none>
I20260926 20:12:59.977958 4302 fs_manager.cc:359] Metadata directory not provided
I20260926 20:12:59.978065 4302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:12:59 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453579, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:13:00.005008 4302 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260926 20:13:00.008407 4318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:00.010201 4302 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:13:00.010480 4302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "eaf0ed99d2804a33bff3866533eb46c8"
format_stamp: "Formatted at 2026-09-26 20:12:58 on dist-test-slave-7034"
server_key: "90c2cc8d3da031ccc8414f16ebb7efbb"
server_key_iv: "d1e6a1f1c0224a5bc4ff1e673f83b1ef"
server_key_version: "kuduclusterkey@0"
I20260926 20:13:00.011270 4302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:00.043766 4302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:00.047724 4302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:00.047999 4302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:00.048861 4302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:00.050083 4302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:13:00.050201 4302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:00.050310 4302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:00.050385 4302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:00.063060 4302 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:42061
I20260926 20:13:00.063215 4432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:42061 every 8 connection(s)
I20260926 20:13:00.064582 4302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:13:00.073081 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 4302
I20260926 20:13:00.073652 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.50.148:42585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:42035
--local_ip_for_outbound_sockets=127.23.50.131
--tserver_master_addrs=127.23.50.190:38107
--webserver_port=39149
--webserver_interface=127.23.50.131
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.50.148:42585/kms with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:13:00 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453579, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:00.081756 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:00.067569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:48681 (local address 127.23.50.190:38107)
0926 20:13:00.067711 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:00.067715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:00.067960 (+ 245us) server_negotiation.cc:407] Connection header received
0926 20:13:00.068982 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:00.068985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:00.069044 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:00.069155 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:00.070431 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:00.070966 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:00.072227 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:00.072594 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:00.072694 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:00.077204 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:00.077208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:00.077211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:00.077242 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:00.079112 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:00.079820 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:00.079823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:00.079825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:00.079922 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:00.080353 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:00.080357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:00.080359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:00.080610 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:00.080689 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:00.081415 (+ 726us) server_negotiation.cc:299] Negotiation successful
0926 20:13:00.081554 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:13:00.083636 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:00.067252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:38107 (local address 127.23.50.130:48681)
0926 20:13:00.067801 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:00.067853 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:00.068759 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:00.069374 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:00.069381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:00.069646 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:00.070229 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:00.070299 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:00.071111 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:00.071113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:00.072081 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:00.072092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:00.074233 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:00.074237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:00.074442 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:00.075156 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:00.075177 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:00.076968 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:00.079271 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:00.079278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:00.079298 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:00.079696 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:00.080043 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:00.080046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:00.080048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:00.080235 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:00.080774 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:00.080781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:00.081119 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 20:13:00.082130 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0926 20:13:00.082501 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":171,"threads_started":1}
I20260926 20:13:00.085322 4433 heartbeater.cc:347] Connected to a master server at 127.23.50.190:38107
I20260926 20:13:00.085573 4433 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:00.086200 4433 heartbeater.cc:511] Master 127.23.50.190:38107 requested a full tablet report, sending...
W20260926 20:13:00.087918 4433 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:38107 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 20:13:00.088043 4433 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:00.088239 4433 heartbeater.cc:511] Master 127.23.50.190:38107 requested a full tablet report, sending...
I20260926 20:13:00.089087 4433 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:00.089241 4433 heartbeater.cc:511] Master 127.23.50.190:38107 requested a full tablet report, sending...
W20260926 20:13:00.090044 4433 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:13:00.225559 4436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:00.225906 4436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:00.226022 4436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:00.230765 4436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:00.230862 4436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:00.231014 4436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:13:00.237813 4436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.50.148:42585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:42035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=39149
--tserver_master_addrs=127.23.50.190:38107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:00.239068 4436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:00.240142 4436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:00.251256 4444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:00.251689 4441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:00.251274 4442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:00.252308 4436 server_base.cc:1034] running on GCE node
I20260926 20:13:00.252841 4436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:00.253741 4436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:00.254914 4436 hybrid_clock.cc:648] HybridClock initialized: now 1790453580254875 us; error 60 us; skew 500 ppm
Sep 26 20:13:00 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:00.258703 4436 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:13:00.259894 4436 webserver.cc:460] Webserver started at http://127.23.50.131:39149/ using document root <none> and password file <none>
I20260926 20:13:00.260651 4436 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:00.260766 4436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 20:13:00 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:13:00.290984 4436 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260926 20:13:00.298810 4452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:00.300813 4436 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20260926 20:13:00.300971 4436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "48c24133df504b22a8c2c76187a1da05"
format_stamp: "Formatted at 2026-09-26 20:12:59 on dist-test-slave-7034"
server_key: "8d593721b7484b04fe4eea1c3b1a7282"
server_key_iv: "0a893fe57b710ad5505a9c558da87571"
server_key_version: "kuduclusterkey@0"
I20260926 20:13:00.301765 4436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:00.331157 4436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:00.334841 4436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:00.335060 4436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:00.335667 4436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:00.336586 4436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:13:00.336656 4436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:00.336723 4436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:00.336777 4436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:00.348228 4436 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:42035
I20260926 20:13:00.348286 4572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:42035 every 8 connection(s)
I20260926 20:13:00.349612 4436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:13:00.353971 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 4436
Sep 26 20:13:00 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:00.365805 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:00.354715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.23.50.190:38107)
0926 20:13:00.356051 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:00.356059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:00.356070 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:13:00.356103 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:00.356105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:00.356146 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:00.356230 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:00.357187 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:00.357933 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:00.359453 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:00.359783 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:00.359904 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:00.362217 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:00.362221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:00.362223 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:00.362273 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:00.364192 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:00.364575 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:00.364581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:00.364583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:00.364666 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:00.364946 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:00.364951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:00.364953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:00.365129 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:00.365342 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:00.365480 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 20:13:00.365599 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1264,"thread_start_us":120,"threads_started":1}
I20260926 20:13:00.366510 4300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:00.351800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:44135 (local address 127.23.50.190:38107)
0926 20:13:00.351947 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:00.351950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:00.353025 (+ 1075us) server_negotiation.cc:407] Connection header received
0926 20:13:00.354054 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:00.354058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:00.354148 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:00.354223 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:00.355565 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:00.356354 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:00.357761 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:00.358142 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:00.358271 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:00.360867 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:00.360871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:00.360874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:00.360901 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:00.362720 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:00.363340 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:00.363344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:00.363345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:00.363428 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:00.363866 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:00.363869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:00.363871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:00.364102 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:00.364203 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:00.366228 (+ 2025us) server_negotiation.cc:299] Negotiation successful
0926 20:13:00.366361 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:13:00.367139 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:00.352318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:38107 (local address 127.23.50.131:44135)
0926 20:13:00.352867 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:00.352911 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:00.353785 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:00.354421 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:00.354431 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:00.354828 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:00.355412 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:00.355427 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:00.356476 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:00.356480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:00.357625 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:00.357635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:00.358317 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:00.358321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:00.358530 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:00.359217 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:00.359240 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:00.360706 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:00.362852 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:00.362861 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:00.362883 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:00.363249 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:00.363520 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:00.363525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:00.363529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:00.363746 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:00.364229 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:00.364236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:00.364583 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 20:13:00.364811 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 20:13:00.365173 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":160,"threads_started":1}
I20260926 20:13:00.368289 4573 heartbeater.cc:347] Connected to a master server at 127.23.50.190:38107
I20260926 20:13:00.368533 4573 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:00.369035 4573 heartbeater.cc:511] Master 127.23.50.190:38107 requested a full tablet report, sending...
I20260926 20:13:00.370543 4100 ts_manager.cc:194] Registered new tserver with Master: 48c24133df504b22a8c2c76187a1da05 (127.23.50.131:42035)
I20260926 20:13:00.372078 4100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:44135
Sep 26 20:13:00 dist-test-slave-7034 krb5kdc[3104](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790453580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:00.798607 4298 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:00.798938 4298 heartbeater.cc:511] Master 127.23.50.190:38107 requested a full tablet report, sending...
I20260926 20:13:00.799774 4100 ts_manager.cc:194] Registered new tserver with Master: 15e73738ee324d1ebb9717143e0241df (127.23.50.129:41537)
I20260926 20:13:00.800583 4100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:60511
Sep 26 20:13:01 dist-test-slave-7034 krb5kdc[3104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for HTTP/127.23.50.148@KRBTEST.COM
I20260926 20:13:01.090253 4433 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:01.090560 4433 heartbeater.cc:511] Master 127.23.50.190:38107 requested a full tablet report, sending...
I20260926 20:13:01.091441 4100 ts_manager.cc:194] Registered new tserver with Master: eaf0ed99d2804a33bff3866533eb46c8 (127.23.50.130:42061)
I20260926 20:13:01.092221 4100 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:48681
I20260926 20:13:01.095654 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:13:01.107791 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.097483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.23.50.190:38107)
0926 20:13:01.098033 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:01.098037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:01.098050 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:13:01.098094 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:01.098096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:01.098137 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:01.098231 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:01.098919 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.099656 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.100686 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.100980 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.101086 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:01.103150 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:01.103154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:01.103156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:01.103192 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:01.105125 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:01.105635 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:01.105642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:01.105647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:01.106105 (+ 458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:01.106418 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:01.106422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:01.106424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:01.106644 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:01.106776 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:01.107480 (+ 704us) server_negotiation.cc:299] Negotiation successful
0926 20:13:01.107613 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":89,"threads_started":1}
I20260926 20:13:01.114359 4100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:13:01.118459 4100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:13:01.152925 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.145930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:42035 (local address 127.0.0.1:50960)
0926 20:13:01.146819 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:01.146871 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:01.147005 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:01.147954 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:01.147960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:01.147984 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:01.148148 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:01.148154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.149761 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.149764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:01.151219 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:01.151231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.152389 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.152393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:01.152513 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:01.152559 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:13:01.152646 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:13:01.152768 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":108,"threads_started":1}
I20260926 20:13:01.154165 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.146369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:41537 (local address 127.0.0.1:36678)
0926 20:13:01.148538 (+ 2169us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:01.148554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:01.148933 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:01.149623 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:01.149625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:01.149643 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:01.149742 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:01.149748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.151250 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.151253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:01.152350 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:01.152362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.153663 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.153666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:01.153769 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:01.153793 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:13:01.153939 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 20:13:01.154019 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2111,"thread_start_us":93,"threads_started":1}
I20260926 20:13:01.154732 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.146100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.23.50.129:41537)
0926 20:13:01.147297 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:01.147304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:01.148645 (+ 1341us) server_negotiation.cc:407] Connection header received
0926 20:13:01.149095 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:01.149101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:01.149324 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:01.149448 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:01.149904 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.151096 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.152498 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.153321 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.153403 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:01.153552 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:01.154466 (+ 914us) server_negotiation.cc:299] Negotiation successful
0926 20:13:01.154584 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":96,"threads_started":1}
I20260926 20:13:01.156380 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.146810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50960 (local address 127.23.50.131:42035)
0926 20:13:01.147462 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:01.147467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:01.147489 (+ 22us) server_negotiation.cc:407] Connection header received
0926 20:13:01.147564 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:01.147569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:01.147782 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:01.148318 (+ 536us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:01.148445 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.149623 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.151384 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.152247 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.155899 (+ 3652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:01.156072 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:01.156123 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 20:13:01.156239 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":90,"threads_started":1}
I20260926 20:13:01.158609 4231 tablet_service.cc:1467] Processing CreateTablet for tablet d9c1e886014c463baf780e194679a041 (DEFAULT_TABLE table=test-table [id=34194d55d65242a782ab91f38ac312a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:01.159543 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.147560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:42061 (local address 127.0.0.1:40462)
0926 20:13:01.153546 (+ 5986us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:01.153560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:01.153635 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:01.154363 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:01.154367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:01.154383 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:01.154454 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:01.154459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.156643 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.156647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:01.157828 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:01.157839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.159138 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.159142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:01.159246 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:01.159270 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:13:01.159340 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:13:01.159411 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5929,"thread_start_us":89,"threads_started":1}
I20260926 20:13:01.159970 4231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c1e886014c463baf780e194679a041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:01.160452 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.147986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40462 (local address 127.23.50.130:42061)
0926 20:13:01.148846 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:01.148854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:01.153734 (+ 4880us) server_negotiation.cc:407] Connection header received
0926 20:13:01.153820 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:01.153825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:01.154075 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:01.154222 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:01.154576 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.155736 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.158011 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.158843 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.158944 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:01.159077 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:01.160211 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0926 20:13:01.160316 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":106,"threads_started":1}
I20260926 20:13:01.163476 4363 tablet_service.cc:1467] Processing CreateTablet for tablet d9c1e886014c463baf780e194679a041 (DEFAULT_TABLE table=test-table [id=34194d55d65242a782ab91f38ac312a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:01.163476 4507 tablet_service.cc:1467] Processing CreateTablet for tablet d9c1e886014c463baf780e194679a041 (DEFAULT_TABLE table=test-table [id=34194d55d65242a782ab91f38ac312a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:01.164587 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c1e886014c463baf780e194679a041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:01.164585 4363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9c1e886014c463baf780e194679a041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:01.171569 4599 tablet_bootstrap.cc:492] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df: Bootstrap starting.
I20260926 20:13:01.173457 4601 tablet_bootstrap.cc:492] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05: Bootstrap starting.
I20260926 20:13:01.174055 4599 tablet_bootstrap.cc:654] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:01.175105 4599 log.cc:824] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:01.175700 4601 tablet_bootstrap.cc:654] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:01.176604 4601 log.cc:824] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:01.177977 4599 tablet_bootstrap.cc:492] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df: No bootstrap required, opened a new log
I20260926 20:13:01.178179 4599 ts_tablet_manager.cc:1397] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:13:01.178670 4601 tablet_bootstrap.cc:492] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05: No bootstrap required, opened a new log
I20260926 20:13:01.178835 4601 ts_tablet_manager.cc:1397] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:13:01.182283 4600 tablet_bootstrap.cc:492] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8: Bootstrap starting.
I20260926 20:13:01.182337 4599 raft_consensus.cc:348] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e73738ee324d1ebb9717143e0241df" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41537 } } peers { permanent_uuid: "48c24133df504b22a8c2c76187a1da05" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42035 } } peers { permanent_uuid: "eaf0ed99d2804a33bff3866533eb46c8" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42061 } }
I20260926 20:13:01.182634 4599 raft_consensus.cc:374] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:01.182684 4599 raft_consensus.cc:729] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e73738ee324d1ebb9717143e0241df, State: Initialized, Role: FOLLOWER
I20260926 20:13:01.182938 4601 raft_consensus.cc:348] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e73738ee324d1ebb9717143e0241df" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41537 } } peers { permanent_uuid: "48c24133df504b22a8c2c76187a1da05" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42035 } } peers { permanent_uuid: "eaf0ed99d2804a33bff3866533eb46c8" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42061 } }
I20260926 20:13:01.183380 4601 raft_consensus.cc:374] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:01.183491 4601 raft_consensus.cc:729] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c24133df504b22a8c2c76187a1da05, State: Initialized, Role: FOLLOWER
I20260926 20:13:01.183456 4599 consensus_queue.cc:260] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e73738ee324d1ebb9717143e0241df" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41537 } } peers { permanent_uuid: "48c24133df504b22a8c2c76187a1da05" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42035 } } peers { permanent_uuid: "eaf0ed99d2804a33bff3866533eb46c8" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42061 } }
I20260926 20:13:01.184005 4601 consensus_queue.cc:260] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e73738ee324d1ebb9717143e0241df" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41537 } } peers { permanent_uuid: "48c24133df504b22a8c2c76187a1da05" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42035 } } peers { permanent_uuid: "eaf0ed99d2804a33bff3866533eb46c8" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42061 } }
I20260926 20:13:01.184705 4599 ts_tablet_manager.cc:1428] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:13:01.184804 4600 tablet_bootstrap.cc:654] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:01.185114 4298 heartbeater.cc:503] Master 127.23.50.190:38107 was elected leader, sending a full tablet report...
I20260926 20:13:01.186857 4600 log.cc:824] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:01.189529 4601 ts_tablet_manager.cc:1428] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260926 20:13:01.189740 4600 tablet_bootstrap.cc:492] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8: No bootstrap required, opened a new log
I20260926 20:13:01.189996 4600 ts_tablet_manager.cc:1397] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 20:13:01.190012 4573 heartbeater.cc:503] Master 127.23.50.190:38107 was elected leader, sending a full tablet report...
I20260926 20:13:01.194279 4600 raft_consensus.cc:348] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e73738ee324d1ebb9717143e0241df" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41537 } } peers { permanent_uuid: "48c24133df504b22a8c2c76187a1da05" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42035 } } peers { permanent_uuid: "eaf0ed99d2804a33bff3866533eb46c8" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42061 } }
I20260926 20:13:01.194546 4600 raft_consensus.cc:374] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:01.194633 4600 raft_consensus.cc:729] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf0ed99d2804a33bff3866533eb46c8, State: Initialized, Role: FOLLOWER
I20260926 20:13:01.195111 4600 consensus_queue.cc:260] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e73738ee324d1ebb9717143e0241df" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41537 } } peers { permanent_uuid: "48c24133df504b22a8c2c76187a1da05" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42035 } } peers { permanent_uuid: "eaf0ed99d2804a33bff3866533eb46c8" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42061 } }
I20260926 20:13:01.195794 4433 heartbeater.cc:503] Master 127.23.50.190:38107 was elected leader, sending a full tablet report...
I20260926 20:13:01.197341 4600 ts_tablet_manager.cc:1428] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 20:13:01.207417 4604 raft_consensus.cc:482] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:01.207692 4604 raft_consensus.cc:504] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e73738ee324d1ebb9717143e0241df" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41537 } } peers { permanent_uuid: "48c24133df504b22a8c2c76187a1da05" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42035 } } peers { permanent_uuid: "eaf0ed99d2804a33bff3866533eb46c8" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42061 } }
I20260926 20:13:01.209254 4604 leader_election.cc:288] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48c24133df504b22a8c2c76187a1da05 (127.23.50.131:42035), eaf0ed99d2804a33bff3866533eb46c8 (127.23.50.130:42061)
I20260926 20:13:01.215186 4598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.209883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:46453 (local address 127.23.50.130:42061)
0926 20:13:01.210022 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:01.210029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:01.210767 (+ 738us) server_negotiation.cc:407] Connection header received
0926 20:13:01.210972 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:01.210975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:01.211025 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:01.211087 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:01.211479 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.212197 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.213632 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.214283 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.214376 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:01.214435 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:01.214962 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 20:13:01.215045 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:13:01.215186 4596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.210169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:41137 (local address 127.23.50.131:42035)
0926 20:13:01.210281 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:01.210286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:01.210352 (+ 66us) server_negotiation.cc:407] Connection header received
0926 20:13:01.210534 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:01.210538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:01.210606 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:01.210680 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:01.211171 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.211977 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.213636 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.214276 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.214376 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:01.214432 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:01.214962 (+ 530us) server_negotiation.cc:299] Negotiation successful
0926 20:13:01.215045 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:13:01.215670 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.209758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:42061 (local address 127.23.50.129:46453)
0926 20:13:01.210689 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:01.210702 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:01.210830 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:01.211231 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:01.211234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:01.211250 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:01.211355 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:01.211360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.212319 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.212322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:01.213489 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:01.213494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.214534 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.214538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:01.214782 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:01.214843 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 20:13:01.215406 (+ 563us) client_negotiation.cc:240] Negotiation successful
0926 20:13:01.215504 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":110,"thread_start_us":142,"threads_started":1}
I20260926 20:13:01.216051 4609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.209266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:42035 (local address 127.23.50.129:41137)
0926 20:13:01.210257 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:01.210284 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:01.210408 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:01.210814 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:01.210817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:01.210875 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:01.211037 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:01.211047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.212112 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.212115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:01.213480 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:01.213492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.214534 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.214538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:01.214786 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:01.214837 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 20:13:01.215687 (+ 850us) client_negotiation.cc:240] Negotiation successful
0926 20:13:01.215844 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":86,"spinlock_wait_cycles":21248,"thread_start_us":165,"threads_started":1}
I20260926 20:13:01.216485 4372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c1e886014c463baf780e194679a041" candidate_uuid: "15e73738ee324d1ebb9717143e0241df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf0ed99d2804a33bff3866533eb46c8" is_pre_election: true
I20260926 20:13:01.216840 4372 raft_consensus.cc:2454] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e73738ee324d1ebb9717143e0241df in term 0.
I20260926 20:13:01.216810 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c1e886014c463baf780e194679a041" candidate_uuid: "15e73738ee324d1ebb9717143e0241df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c24133df504b22a8c2c76187a1da05" is_pre_election: true
I20260926 20:13:01.217087 4527 raft_consensus.cc:2454] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15e73738ee324d1ebb9717143e0241df in term 0.
I20260926 20:13:01.217587 4186 leader_election.cc:302] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e73738ee324d1ebb9717143e0241df, 48c24133df504b22a8c2c76187a1da05; no voters:
I20260926 20:13:01.217880 4604 raft_consensus.cc:2781] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:13:01.217969 4604 raft_consensus.cc:482] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:13:01.218014 4604 raft_consensus.cc:3037] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:01.219133 4604 raft_consensus.cc:504] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e73738ee324d1ebb9717143e0241df" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41537 } } peers { permanent_uuid: "48c24133df504b22a8c2c76187a1da05" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42035 } } peers { permanent_uuid: "eaf0ed99d2804a33bff3866533eb46c8" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42061 } }
I20260926 20:13:01.219537 4604 leader_election.cc:288] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [CANDIDATE]: Term 1 election: Requested vote from peers 48c24133df504b22a8c2c76187a1da05 (127.23.50.131:42035), eaf0ed99d2804a33bff3866533eb46c8 (127.23.50.130:42061)
I20260926 20:13:01.219939 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c1e886014c463baf780e194679a041" candidate_uuid: "15e73738ee324d1ebb9717143e0241df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c24133df504b22a8c2c76187a1da05"
I20260926 20:13:01.219933 4372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c1e886014c463baf780e194679a041" candidate_uuid: "15e73738ee324d1ebb9717143e0241df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf0ed99d2804a33bff3866533eb46c8"
I20260926 20:13:01.220060 4527 raft_consensus.cc:3037] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:01.220060 4372 raft_consensus.cc:3037] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:01.221356 4527 raft_consensus.cc:2454] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e73738ee324d1ebb9717143e0241df in term 1.
I20260926 20:13:01.221356 4372 raft_consensus.cc:2454] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15e73738ee324d1ebb9717143e0241df in term 1.
I20260926 20:13:01.221680 4186 leader_election.cc:302] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e73738ee324d1ebb9717143e0241df, eaf0ed99d2804a33bff3866533eb46c8; no voters:
I20260926 20:13:01.221909 4604 raft_consensus.cc:2781] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:13:01.222155 4604 raft_consensus.cc:686] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [term 1 LEADER]: Becoming Leader. State: Replica: 15e73738ee324d1ebb9717143e0241df, State: Running, Role: LEADER
I20260926 20:13:01.222543 4604 consensus_queue.cc:237] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e73738ee324d1ebb9717143e0241df" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41537 } } peers { permanent_uuid: "48c24133df504b22a8c2c76187a1da05" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42035 } } peers { permanent_uuid: "eaf0ed99d2804a33bff3866533eb46c8" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42061 } }
I20260926 20:13:01.225970 4100 catalog_manager.cc:5495] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df reported cstate change: term changed from 0 to 1, leader changed from <none> to 15e73738ee324d1ebb9717143e0241df (127.23.50.129). New cstate: current_term: 1 leader_uuid: "15e73738ee324d1ebb9717143e0241df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e73738ee324d1ebb9717143e0241df" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48c24133df504b22a8c2c76187a1da05" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf0ed99d2804a33bff3866533eb46c8" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42061 } health_report { overall_health: UNKNOWN } } }
I20260926 20:13:01.254019 4594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.249150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.23.50.129:41537)
0926 20:13:01.249295 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:01.249300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:01.249315 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:13:01.249404 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:01.249407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:01.249470 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:01.249583 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:13:01.249931 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.250725 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.251997 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:01.252344 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:01.252441 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:01.252892 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:13:01.253522 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:13:01.253659 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:01.253744 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:13:01.253815 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:13:01.261746 4527 raft_consensus.cc:1261] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05 [term 1 FOLLOWER]: Refusing update from remote peer 15e73738ee324d1ebb9717143e0241df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:13:01.261713 4372 raft_consensus.cc:1261] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8 [term 1 FOLLOWER]: Refusing update from remote peer 15e73738ee324d1ebb9717143e0241df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:13:01.262562 4610 consensus_queue.cc:1035] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [LEADER]: Connected to new peer: Peer: permanent_uuid: "48c24133df504b22a8c2c76187a1da05" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:13:01.262828 4604 consensus_queue.cc:1035] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf0ed99d2804a33bff3866533eb46c8" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 20:13:01.265201 4299 tablet.cc:2369] T d9c1e886014c463baf780e194679a041 P 15e73738ee324d1ebb9717143e0241df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:13:01.279067 4614 mvcc.cc:205] Tried to move back new op lower bound from 7333697868839137280 to 7333697868690952192. Current Snapshot: MvccSnapshot[applied={T|T < 7333697868839137280}]
I20260926 20:13:01.279392 4615 mvcc.cc:205] Tried to move back new op lower bound from 7333697868839137280 to 7333697868690952192. Current Snapshot: MvccSnapshot[applied={T|T < 7333697868839137280}]
I20260926 20:13:01.288558 4099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:13:01.288762 4099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:13:01.291733 4099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb756fdee4654a909d2de5b2be276726: Sending DeleteTablet for 3 replicas of tablet d9c1e886014c463baf780e194679a041
I20260926 20:13:01.293417 4231 tablet_service.cc:1514] Processing DeleteTablet for tablet d9c1e886014c463baf780e194679a041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:01 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:36678
I20260926 20:13:01.294018 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet d9c1e886014c463baf780e194679a041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:01 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:50960
I20260926 20:13:01.294581 4363 tablet_service.cc:1514] Processing DeleteTablet for tablet d9c1e886014c463baf780e194679a041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:01 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:40462
I20260926 20:13:01.295866 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 4138
I20260926 20:13:01.298441 4623 tablet_replica.cc:330] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05: stopping tablet replica
I20260926 20:13:01.298976 4623 raft_consensus.cc:2229] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:13:01.299448 4623 raft_consensus.cc:2258] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:01.299943 4624 tablet_replica.cc:330] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8: stopping tablet replica
I20260926 20:13:01.300351 4624 raft_consensus.cc:2229] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:13:01.300710 4624 raft_consensus.cc:2258] T d9c1e886014c463baf780e194679a041 P eaf0ed99d2804a33bff3866533eb46c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:01.302386 4623 ts_tablet_manager.cc:1905] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:01.305248 4623 ts_tablet_manager.cc:1918] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:13:01.305470 4623 log.cc:1196] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/d9c1e886014c463baf780e194679a041
W20260926 20:13:01.305776 4087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:41537 (error 108)
I20260926 20:13:01.305898 4623 ts_tablet_manager.cc:1939] T d9c1e886014c463baf780e194679a041 P 48c24133df504b22a8c2c76187a1da05: Deleting consensus metadata
I20260926 20:13:01.306753 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 4302
I20260926 20:13:01.308993 4085 catalog_manager.cc:4841] TS 48c24133df504b22a8c2c76187a1da05 (127.23.50.131:42035): tablet d9c1e886014c463baf780e194679a041 (table test-table [id=34194d55d65242a782ab91f38ac312a3]) successfully deleted
I20260926 20:13:01.309439 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.309006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:41537 (local address 127.0.0.1:36694)
0926 20:13:01.309268 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:01.309348 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:41537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
W20260926 20:13:01.309616 4087 catalog_manager.cc:4584] TS 15e73738ee324d1ebb9717143e0241df (127.23.50.129:41537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9c1e886014c463baf780e194679a041: Network error: Client connection negotiation failed: client connection to 127.23.50.129:41537: connect: Connection refused (error 111)
I20260926 20:13:01.314303 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 4436
I20260926 20:13:01.314783 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:01.314518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:42061 (local address 127.0.0.1:40478)
0926 20:13:01.314667 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:01.314709 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.130:42061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 20:13:01.321033 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 4067
I20260926 20:13:01.328081 23754 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 20:13:01.662847 23754 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 20:13:01.662993 23754 mini_ranger.cc:66] Stopping Ranger...
I20260926 20:13:01.999365 23754 mini_ranger.cc:68] Stopped Ranger
2026-09-26 20:13:01.999 UTC [3134] LOG: received smart shutdown request
2026-09-26 20:13:02.003 UTC [3134] LOG: background worker "logical replication launcher" (PID 3143) exited with exit code 1
2026-09-26 20:13:02.004 UTC [3138] LOG: shutting down
2026-09-26 20:13:02.184 UTC [3134] LOG: database system is shut down
2026-09-26T20:13:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49049 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:13:02 dist-test-slave-7034 krb5kdc[4634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:13:02 dist-test-slave-7034 krb5kdc[4634](info): set up 2 sockets
Sep 26 20:13:02 dist-test-slave-7034 krb5kdc[4634](info): commencing operation
krb5kdc: starting...
W20260926 20:13:03.390447 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:13:03 dist-test-slave-7034 krb5kdc[4634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:13:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:13:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:13:03.512876 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:43613
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:43613,127.23.50.189:37433,127.23.50.188:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:03.625834 4647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:03.626135 4647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:03.626178 4647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:03.630357 4647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:13:03.630424 4647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:03.630443 4647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:03.630460 4647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:13:03.630476 4647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:13:03.635834 4647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:43613,127.23.50.189:37433,127.23.50.188:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:43613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:03.636901 4647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:03.637969 4647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:03.646761 4653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:03.646732 4652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:03.646747 4647 server_base.cc:1034] running on GCE node
W20260926 20:13:03.646775 4655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:03.647535 4647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:03.648645 4647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:03.649816 4647 hybrid_clock.cc:648] HybridClock initialized: now 1790453583649797 us; error 31 us; skew 500 ppm
Sep 26 20:13:03 dist-test-slave-7034 krb5kdc[4634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453583, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:03.657397 4647 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:13:03.658351 4647 webserver.cc:460] Webserver started at http://127.23.50.190:45689/ using document root <none> and password file <none>
I20260926 20:13:03.658900 4647 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:03.658959 4647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:03.659128 4647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:13:03.661116 4647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "dd96985ec212447c9c28ff81f94f7368"
format_stamp: "Formatted at 2026-09-26 20:13:03 on dist-test-slave-7034"
server_key: "af4bacfde46e0a9f7e530aa514b2c61b"
server_key_iv: "d5f9107247a366649736ec6a8961aa30"
server_key_version: "encryptionkey@0"
I20260926 20:13:03.661623 4647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "dd96985ec212447c9c28ff81f94f7368"
format_stamp: "Formatted at 2026-09-26 20:13:03 on dist-test-slave-7034"
server_key: "af4bacfde46e0a9f7e530aa514b2c61b"
server_key_iv: "d5f9107247a366649736ec6a8961aa30"
server_key_version: "encryptionkey@0"
I20260926 20:13:03.665269 4647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 20:13:03.668540 4662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:03.670257 4647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 20:13:03.670421 4647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "dd96985ec212447c9c28ff81f94f7368"
format_stamp: "Formatted at 2026-09-26 20:13:03 on dist-test-slave-7034"
server_key: "af4bacfde46e0a9f7e530aa514b2c61b"
server_key_iv: "d5f9107247a366649736ec6a8961aa30"
server_key_version: "encryptionkey@0"
I20260926 20:13:03.670569 4647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:03.686290 4647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:03.689332 4647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:03.689538 4647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:03.696789 4647 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:43613
I20260926 20:13:03.696817 4714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:43613 every 8 connection(s)
I20260926 20:13:03.697788 4647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:13:03.699282 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 4647
I20260926 20:13:03.699440 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:13:03.699734 23754 external_mini_cluster.cc:1428] Setting key 856186d7ce4420b55479208f3e98ec31
I20260926 20:13:03.700829 4715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:03.705535 4715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.50.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.189@KRBTEST.COM" created.
Sep 26 20:13:03 dist-test-slave-7034 krb5kdc[4634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453583, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:03.724026 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.708517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43613 (local address 127.0.0.1:37418)
0926 20:13:03.709013 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.709063 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:03.709997 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:03.710825 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:03.710834 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:03.711331 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:03.711606 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:03.711620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.713328 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.713334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:03.714683 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:03.714698 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.715494 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.715501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:03.715651 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:03.716279 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:03.716316 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:03.718338 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:03.720445 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:03.720454 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:03.720463 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:03.720731 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:03.721345 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:03.721347 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:03.721349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:03.721470 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:03.722331 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:03.722337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:03.722486 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 20:13:03.722654 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 20:13:03.723028 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":171,"threads_started":1}
I20260926 20:13:03.724032 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.708517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.23.50.190:43613)
0926 20:13:03.708964 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:03.708978 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:03.709231 (+ 253us) server_negotiation.cc:407] Connection header received
0926 20:13:03.710337 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:03.710342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:03.710581 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:03.710774 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:03.711781 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.713167 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.714824 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.715314 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.715445 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:03.718527 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:03.718533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:03.718544 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:03.718567 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:03.720334 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:03.720874 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:03.720879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:03.720880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:03.721092 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:03.721603 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:03.721606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:03.721607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:03.722194 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:03.722349 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:03.722838 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 20:13:03.723133 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":290,"thread_start_us":150,"threads_started":1}
I20260926 20:13:03.726815 4715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:13:03.727780 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.727374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:37433 (local address 127.0.0.1:35630)
0926 20:13:03.727576 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.727709 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 20:13:03.728005 4666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.50.189:37433: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.189.keytab.
Entry for principal kudu/127.23.50.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.189.keytab.
I20260926 20:13:03.728996 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.728731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:37433 (local address 127.0.0.1:35636)
0926 20:13:03.728854 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.728919 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 20:13:03.729148 4715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.189:37433: Network error: Client connection negotiation failed: client connection to 127.23.50.189:37433: connect: Connection refused (error 111)
I20260926 20:13:03.729242 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.189:37433
--webserver_interface=127.23.50.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.189.keytab
--principal=kudu/127.23.50.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:43613,127.23.50.189:37433,127.23.50.188:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:13:03.778426 4715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } attempt: 1
I20260926 20:13:03.779331 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.778992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:37433 (local address 127.0.0.1:35640)
0926 20:13:03.779162 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.779237 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 20:13:03.780077 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.779849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:37433 (local address 127.0.0.1:35652)
0926 20:13:03.779962 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.780013 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 20:13:03.780225 4715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.189:37433: Network error: Client connection negotiation failed: client connection to 127.23.50.189:37433: connect: Connection refused (error 111)
W20260926 20:13:03.839532 4722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:03.839800 4722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:03.839840 4722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:03.843984 4722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:13:03.844044 4722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:03.844064 4722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:03.844081 4722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:13:03.844096 4722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:13:03.848387 4715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } attempt: 2
I20260926 20:13:03.849189 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.848882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:37433 (local address 127.0.0.1:35668)
0926 20:13:03.849037 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.849105 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 20:13:03.849921 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.849689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:37433 (local address 127.0.0.1:35676)
0926 20:13:03.849788 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.849838 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:37433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 20:13:03.850047 4715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.189:37433: Network error: Client connection negotiation failed: client connection to 127.23.50.189:37433: connect: Connection refused (error 111)
I20260926 20:13:03.850502 4722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:43613,127.23.50.189:37433,127.23.50.188:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.189.keytab
--principal=kudu/127.23.50.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.189:37433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.50.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:03.851539 4722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:03.852491 4722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:03.860869 4730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:03.860865 4727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:03.860863 4728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:03.861239 4722 server_base.cc:1034] running on GCE node
I20260926 20:13:03.861795 4722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:03.862934 4722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:03.864097 4722 hybrid_clock.cc:648] HybridClock initialized: now 1790453583864064 us; error 47 us; skew 500 ppm
Sep 26 20:13:03 dist-test-slave-7034 krb5kdc[4634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453583, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:03.866930 4722 init.cc:377] Logged in from keytab as kudu/127.23.50.189@KRBTEST.COM (short username kudu)
I20260926 20:13:03.867844 4722 webserver.cc:460] Webserver started at http://127.23.50.189:38697/ using document root <none> and password file <none>
I20260926 20:13:03.868382 4722 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:03.868443 4722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:03.868610 4722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:13:03.870937 4722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/data/instance:
uuid: "464c2847e5b84c7ea5d4c1ec8955928a"
format_stamp: "Formatted at 2026-09-26 20:13:03 on dist-test-slave-7034"
server_key: "9bb1298697454c957b775e1997011642"
server_key_iv: "4cd63c8773599025a27f7f663b3f48ff"
server_key_version: "encryptionkey@0"
I20260926 20:13:03.871536 4722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/wal/instance:
uuid: "464c2847e5b84c7ea5d4c1ec8955928a"
format_stamp: "Formatted at 2026-09-26 20:13:03 on dist-test-slave-7034"
server_key: "9bb1298697454c957b775e1997011642"
server_key_iv: "4cd63c8773599025a27f7f663b3f48ff"
server_key_version: "encryptionkey@0"
I20260926 20:13:03.875447 4722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:13:03.878044 4737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:03.879359 4722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:13:03.879498 4722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/wal
uuid: "464c2847e5b84c7ea5d4c1ec8955928a"
format_stamp: "Formatted at 2026-09-26 20:13:03 on dist-test-slave-7034"
server_key: "9bb1298697454c957b775e1997011642"
server_key_iv: "4cd63c8773599025a27f7f663b3f48ff"
server_key_version: "encryptionkey@0"
I20260926 20:13:03.879609 4722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:03.910291 4722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:03.913471 4722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:03.913687 4722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:03.921139 4722 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.189:37433
I20260926 20:13:03.921221 4789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.189:37433 every 8 connection(s)
I20260926 20:13:03.922210 4722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
I20260926 20:13:03.925220 4790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:03.925977 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 4722
I20260926 20:13:03.926158 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-1/wal/instance
I20260926 20:13:03.926457 23754 external_mini_cluster.cc:1428] Setting key b19b03acbd6f66bf515d7433bd2b3c68
I20260926 20:13:03.929598 4790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.50.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.188@KRBTEST.COM" created.
Sep 26 20:13:03 dist-test-slave-7034 krb5kdc[4634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453583, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.189@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:03.941200 4715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } attempt: 3
I20260926 20:13:03.946736 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.931817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.23.50.190:43613)
0926 20:13:03.932008 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:03.932013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:03.932982 (+ 969us) server_negotiation.cc:407] Connection header received
0926 20:13:03.934050 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:03.934056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:03.934138 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:03.934283 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:03.935558 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.936234 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.937945 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.938191 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.938293 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:03.941541 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:03.941545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:03.941547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:03.941574 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:03.943777 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:03.944348 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:03.944352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:03.944354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:03.944446 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:03.944934 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:03.944945 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:03.944948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:03.945471 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:03.945595 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:03.946404 (+ 809us) server_negotiation.cc:299] Negotiation successful
0926 20:13:03.946544 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":210,"server-negotiator.queue_time_us":70}
I20260926 20:13:03.947891 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.932269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43613 (local address 127.0.0.1:37422)
0926 20:13:03.932783 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.932840 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:03.933756 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:03.934495 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:03.934502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:03.934858 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:03.935389 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:03.935403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.936393 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.936396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:03.937707 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:03.937724 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.938374 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.938381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:03.938645 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:03.939391 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:03.939409 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:03.941319 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:03.943908 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:03.943914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:03.943932 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:03.944231 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:03.944545 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:03.944549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:03.944551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:03.944717 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:03.945631 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:03.945640 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:03.946063 (+ 423us) client_negotiation.cc:769] Sending connection context
0926 20:13:03.946333 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 20:13:03.946668 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":163,"threads_started":1}
Sep 26 20:13:03 dist-test-slave-7034 krb5kdc[4634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453583, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.189@KRBTEST.COM
I20260926 20:13:03.950443 4790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:13:03.953953 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.941715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:37433 (local address 127.0.0.1:35692)
0926 20:13:03.941917 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.941935 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:03.942022 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:03.942906 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:03.942909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:03.943064 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:03.943188 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:03.943194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.944969 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.944972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:03.946066 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:03.946077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.947022 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.947026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:03.947149 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:03.947813 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:03.947826 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:03.949482 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:03.951989 (+ 2507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:03.951993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:03.951995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:03.952209 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:03.952627 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:03.952630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:03.952632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:03.952721 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:03.953282 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:03.953285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:03.953409 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 20:13:03.953524 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 20:13:03.953728 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":21}
I20260926 20:13:03.954380 4795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.941902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.23.50.189:37433)
0926 20:13:03.942344 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:03.942366 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:03.942402 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:13:03.942458 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:03.942466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:03.942752 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:03.942925 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:03.943327 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.944766 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.946245 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.946845 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.947018 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:03.949674 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:03.949681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:03.949696 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:03.949723 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:03.951878 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:03.952335 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:03.952341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:03.952343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:03.952481 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:03.952827 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:03.952830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:03.952832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:03.953173 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:03.953321 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:03.953665 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 20:13:03.953875 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":279,"thread_start_us":157,"threads_started":1}
Sep 26 20:13:03 dist-test-slave-7034 krb5kdc[4634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453583, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.189@KRBTEST.COM for kudu/127.23.50.189@KRBTEST.COM
I20260926 20:13:03.956672 4715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.188.keytab.
Entry for principal kudu/127.23.50.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.188.keytab.
I20260926 20:13:03.957549 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.957205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35334)
0926 20:13:03.957402 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.957470 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 20:13:03.958267 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.958021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35346)
0926 20:13:03.958163 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.958212 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 20:13:03.958328 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.188:32969
--webserver_interface=127.23.50.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.188.keytab
--principal=kudu/127.23.50.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:43613,127.23.50.189:37433,127.23.50.188:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:03.958477 4715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:32969: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
I20260926 20:13:03.960566 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.951121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.23.50.189:37433)
0926 20:13:03.951468 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:03.951475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:03.951485 (+ 10us) server_negotiation.cc:407] Connection header received
0926 20:13:03.951510 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:03.951512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:03.951544 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:03.951642 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:03.952105 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.952661 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.953531 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.953893 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.953989 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:03.956780 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:03.956783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:03.956786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:03.956839 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:03.958774 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:03.959186 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:03.959193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:03.959197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:03.959303 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:03.959558 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:03.959564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:03.959568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:03.959836 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:03.959939 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:03.960237 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 20:13:03.960401 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":283,"thread_start_us":149,"threads_started":1}
I20260926 20:13:03.960596 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.951044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:37433 (local address 127.0.0.1:35700)
0926 20:13:03.951176 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.951193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:03.951281 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:03.951650 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:03.951652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:03.951921 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:03.951998 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:03.952003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.952778 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.952781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:03.953406 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:03.953415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:03.953988 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:03.953992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:03.954074 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:03.954608 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:03.954619 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:03.956630 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:03.958902 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:03.958905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:03.958906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:03.959094 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:03.959397 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:03.959399 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:03.959400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:03.959471 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:03.959947 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:03.959951 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:03.960063 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 20:13:03.960189 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 20:13:03.960349 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 20:13:03.961357 4790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:13:03.962162 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.961800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35350)
0926 20:13:03.961984 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.962078 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260926 20:13:03.962401 4738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
I20260926 20:13:03.963452 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.963176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35360)
0926 20:13:03.963303 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.963368 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 20:13:03.963677 4790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:32969: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
I20260926 20:13:03.989658 4715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } attempt: 1
I20260926 20:13:03.990605 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.990263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35368)
0926 20:13:03.990432 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.990506 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 20:13:03.991366 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:03.991122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35384)
0926 20:13:03.991248 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:03.991301 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 20:13:03.991497 4715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:32969: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
I20260926 20:13:04.012984 4790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } attempt: 1
I20260926 20:13:04.014007 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.013638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35390)
0926 20:13:04.013798 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:04.013923 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 20:13:04.014773 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.014572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35400)
0926 20:13:04.014674 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:04.014720 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:13:04.014943 4790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:32969: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
I20260926 20:13:04.066664 4715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } attempt: 2
I20260926 20:13:04.067556 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.067222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35416)
0926 20:13:04.067393 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:04.067464 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 20:13:04.068349 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.068085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35420)
0926 20:13:04.068197 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:04.068253 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:13:04.068533 4715 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:32969: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
W20260926 20:13:04.073027 4797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:04.073318 4797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:04.073414 4797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:04.077567 4797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:13:04.077642 4797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:04.077688 4797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:04.077728 4797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:13:04.077762 4797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:13:04.083099 4790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } attempt: 2
I20260926 20:13:04.083225 4797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:43613,127.23.50.189:37433,127.23.50.188:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.188.keytab
--principal=kudu/127.23.50.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.188:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.50.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:04.083954 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.083631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35428)
0926 20:13:04.083815 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:04.083879 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 20:13:04.084618 4797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:04.084690 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.084457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35436)
0926 20:13:04.084555 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:04.084622 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:13:04.084858 4790 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:32969: Network error: Client connection negotiation failed: client connection to 127.23.50.188:32969: connect: Connection refused (error 111)
I20260926 20:13:04.085704 4797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:04.093932 4806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:04.093932 4803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:04.094153 4797 server_base.cc:1034] running on GCE node
W20260926 20:13:04.094256 4804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:04.094805 4797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:04.095916 4797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:04.097141 4797 hybrid_clock.cc:648] HybridClock initialized: now 1790453584097131 us; error 47 us; skew 500 ppm
Sep 26 20:13:04 dist-test-slave-7034 krb5kdc[4634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453584, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:04.099997 4797 init.cc:377] Logged in from keytab as kudu/127.23.50.188@KRBTEST.COM (short username kudu)
I20260926 20:13:04.100884 4797 webserver.cc:460] Webserver started at http://127.23.50.188:40417/ using document root <none> and password file <none>
I20260926 20:13:04.101434 4797 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:04.101524 4797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:04.101778 4797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:13:04.103900 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/data/instance:
uuid: "efaba2c9d4784a30bcb49673b834ce15"
format_stamp: "Formatted at 2026-09-26 20:13:04 on dist-test-slave-7034"
server_key: "942d6d4e245969ff99235b0f39c17b71"
server_key_iv: "21163b0eca90342741856509961bd13d"
server_key_version: "encryptionkey@0"
I20260926 20:13:04.104562 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/wal/instance:
uuid: "efaba2c9d4784a30bcb49673b834ce15"
format_stamp: "Formatted at 2026-09-26 20:13:04 on dist-test-slave-7034"
server_key: "942d6d4e245969ff99235b0f39c17b71"
server_key_iv: "21163b0eca90342741856509961bd13d"
server_key_version: "encryptionkey@0"
I20260926 20:13:04.108472 4797 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:13:04.110975 4813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:04.112320 4797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:13:04.112483 4797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/wal
uuid: "efaba2c9d4784a30bcb49673b834ce15"
format_stamp: "Formatted at 2026-09-26 20:13:04 on dist-test-slave-7034"
server_key: "942d6d4e245969ff99235b0f39c17b71"
server_key_iv: "21163b0eca90342741856509961bd13d"
server_key_version: "encryptionkey@0"
I20260926 20:13:04.112607 4797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:04.131137 4797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:04.135046 4797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:04.135254 4797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:04.142675 4797 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.188:32969
I20260926 20:13:04.142694 4865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.188:32969 every 8 connection(s)
I20260926 20:13:04.143630 4797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
I20260926 20:13:04.145891 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 4797
I20260926 20:13:04.146031 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790453421225356-23754-0/minicluster-data/master-2/wal/instance
I20260926 20:13:04.146312 23754 external_mini_cluster.cc:1428] Setting key be0747640e7343d5b309712513eb515b
I20260926 20:13:04.146801 4866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:04.150918 4866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:13:04 dist-test-slave-7034 krb5kdc[4634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.188@KRBTEST.COM
Sep 26 20:13:04 dist-test-slave-7034 krb5kdc[4634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453584, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.188@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:04.163184 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.152909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.23.50.190:43613)
0926 20:13:04.153048 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:04.153054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:04.153366 (+ 312us) server_negotiation.cc:407] Connection header received
0926 20:13:04.153519 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:04.153523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:04.153574 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:04.153664 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:04.154357 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.154851 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.155719 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.156059 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.156178 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.158945 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:04.158951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:04.158955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:04.158988 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:04.160937 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.161557 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.161561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.161563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.161665 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.162076 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.162080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.162082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.162294 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:04.162402 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:04.162884 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 20:13:04.163010 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:13:04.164023 4870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.152852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43613 (local address 127.0.0.1:37426)
0926 20:13:04.153246 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:04.153283 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:04.153418 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:04.153711 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:04.153717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:04.154123 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:04.154237 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:04.154242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.154970 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.154974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:04.155590 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:04.155598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.156209 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.156214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:04.156321 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.156859 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:04.156877 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:04.158741 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:04.161149 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:04.161159 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:04.161170 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:04.161441 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:04.161769 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:04.161772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:04.161774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:04.161977 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:04.162401 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:04.162406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:04.162564 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 20:13:04.162751 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 20:13:04.163210 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":99,"thread_start_us":165,"threads_started":1}
I20260926 20:13:04.164026 4869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.147719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.23.50.188:32969)
0926 20:13:04.148224 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:04.148240 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:04.150216 (+ 1976us) server_negotiation.cc:407] Connection header received
0926 20:13:04.150968 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:04.150986 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:04.151395 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:04.151697 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:04.152588 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.153978 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.154840 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.155337 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.155519 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.158449 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:04.158455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:04.158465 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:04.158488 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:04.160657 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.161315 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.161320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.161321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.161441 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.161766 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.161770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.161772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.162433 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:04.162601 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:04.162877 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 20:13:04.163083 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":134,"threads_started":1}
I20260926 20:13:04.165455 4866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:13:04.167693 4715 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } attempt: 3
Sep 26 20:13:04 dist-test-slave-7034 krb5kdc[4634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453584, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.188@KRBTEST.COM for kudu/127.23.50.189@KRBTEST.COM
Sep 26 20:13:04 dist-test-slave-7034 krb5kdc[4634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
Sep 26 20:13:04 dist-test-slave-7034 krb5kdc[4634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453583, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.188@KRBTEST.COM
I20260926 20:13:04.175388 4870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.166050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:37433 (local address 127.0.0.1:35710)
0926 20:13:04.166239 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:04.166253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:04.166326 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:04.166565 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:04.166568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:04.166742 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:04.166874 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:04.166883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.167972 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.167979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:04.168753 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:04.168760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.169235 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.169239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:04.169328 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.169842 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:04.169878 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:04.171336 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:04.173582 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:04.173588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:04.173593 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:04.173840 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:04.174212 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:04.174216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:04.174218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:04.174380 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:04.174831 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:04.174835 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:04.174936 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 20:13:04.175050 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 20:13:04.175186 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 20:13:04.175781 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.166166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.23.50.189:37433)
0926 20:13:04.166347 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:04.166353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:04.166371 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:13:04.166416 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:04.166419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:04.166465 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:04.166552 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:04.167008 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.167875 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.168871 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.169121 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.169238 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.171478 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:04.171484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:04.171489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:04.171525 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:04.173479 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.173985 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.173989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.173991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.174106 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.174473 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.174479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.174484 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.174715 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:04.174798 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:04.175476 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 20:13:04.175610 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:13:04.175985 4790 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } attempt: 3
I20260926 20:13:04.176820 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.166769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.23.50.190:43613)
0926 20:13:04.166884 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:04.166888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:04.166902 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:13:04.166960 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:04.166963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:04.167006 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:04.167069 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:04.167841 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.168544 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.169667 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.170027 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.170141 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.172777 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:04.172780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:04.172783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:04.172809 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:04.174453 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.174872 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.174875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.174877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.174958 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.175367 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.175371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.175372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.175710 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:04.175935 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:04.176540 (+ 605us) server_negotiation.cc:299] Negotiation successful
0926 20:13:04.176655 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:13:04.177762 4719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.168089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35460)
0926 20:13:04.168202 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:04.168216 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:04.168286 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:04.168584 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:04.168589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:04.168770 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:04.168855 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:04.168861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.170308 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.170314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:04.171160 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:04.171171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.171775 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.171782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:04.171888 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.172406 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:04.172425 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:04.174000 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:04.176118 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:04.176121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:04.176125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:04.176396 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:04.176731 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:04.176737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:04.176742 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:04.176842 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:04.177292 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:04.177296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:04.177386 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 20:13:04.177470 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 20:13:04.177598 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 20:13:04.177973 4866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:13:04.179282 4869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.168279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.23.50.188:32969)
0926 20:13:04.168387 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:04.168391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:04.168403 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:13:04.168433 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:04.168436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:04.168470 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:04.168544 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:04.168983 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.170164 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.171309 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.171647 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.171752 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.174137 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:04.174143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:04.174149 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:04.174177 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:04.175995 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.176520 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.176523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.176524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.176604 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.176956 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.176959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.176960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.177178 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:04.178936 (+ 1758us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:04.178998 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 20:13:04.179144 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 20:13:04 dist-test-slave-7034 krb5kdc[4634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453583, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.189@KRBTEST.COM for kudu/127.23.50.188@KRBTEST.COM
I20260926 20:13:04.182951 4715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368: Bootstrap starting.
Sep 26 20:13:04 dist-test-slave-7034 krb5kdc[4634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453584, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.188@KRBTEST.COM for kudu/127.23.50.188@KRBTEST.COM
Sep 26 20:13:04 dist-test-slave-7034 krb5kdc[4634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.189@KRBTEST.COM
I20260926 20:13:04.186337 4715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:04.187367 4715 log.cc:824] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:04.188578 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.176396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35464)
0926 20:13:04.176632 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:04.176648 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:04.176717 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:04.177517 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:04.177520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:04.177697 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:04.177799 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:04.177809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.178822 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.178826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:04.179623 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:04.179634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.180257 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.180260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:04.180340 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.180820 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:04.180833 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:04.182791 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:04.186866 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:04.186869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:04.186871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:04.187131 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:04.187463 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:04.187466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:04.187467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:04.187545 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:04.188045 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:04.188049 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:04.188155 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 20:13:04.188246 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:13:04.188399 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 20:13:04.188439 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.178295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.23.50.189:37433)
0926 20:13:04.178398 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:04.178402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:04.178479 (+ 77us) server_negotiation.cc:407] Connection header received
0926 20:13:04.178607 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:04.178610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:04.178647 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:04.178716 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:04.179742 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.180463 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.181511 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.181866 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.181938 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.185136 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:04.185140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:04.185142 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:04.185175 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:04.186467 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.186869 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.186873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.186875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.186995 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.187282 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.187285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.187287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.187523 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:04.187755 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:04.188123 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 20:13:04.188275 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":1792}
I20260926 20:13:04.188946 4870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.178376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:32969 (local address 127.0.0.1:35472)
0926 20:13:04.178738 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:04.178752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:04.178816 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:04.179069 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:04.179074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:04.179269 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:04.179381 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:04.179390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.180484 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.180490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:04.181283 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:04.181302 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.181912 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.181915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:04.181984 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.182398 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:04.182411 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:04.183921 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:04.187492 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:04.187494 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:04.187496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:04.187717 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:04.188006 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:04.188011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:04.188014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:04.188116 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:04.188507 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:04.188509 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:04.188592 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 20:13:04.188690 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 20:13:04.188804 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299}
I20260926 20:13:04.188876 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.176956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.23.50.188:32969)
0926 20:13:04.177287 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:04.177295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:04.177308 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:13:04.177341 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:04.177344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:04.177382 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:04.177472 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:04.177956 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.178692 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.179765 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.180108 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.182940 (+ 2832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.183047 (+ 107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:04.183051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:04.183053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:04.183072 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:04.185703 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.187276 (+ 1573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.187279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.187281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.187363 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.187699 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.187703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.187704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.187921 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:04.188510 (+ 589us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:04.188581 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 20:13:04.188700 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":130,"threads_started":1}
I20260926 20:13:04.189244 4872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.178545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.23.50.188:32969)
0926 20:13:04.178845 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:04.178854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:04.178866 (+ 12us) server_negotiation.cc:407] Connection header received
0926 20:13:04.178900 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:04.178904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:04.178944 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:04.179480 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:04.179589 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.180375 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.181420 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.181772 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.182185 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.184063 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:04.184067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:04.184069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:04.184092 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:04.186692 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.187829 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.187831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.187833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.187911 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.188215 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.188221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.188225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.188415 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:04.188512 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:04.188959 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 20:13:04.189063 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":97,"threads_started":1}
I20260926 20:13:04.192288 4790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a: Bootstrap starting.
I20260926 20:13:04.192288 4866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15: Bootstrap starting.
I20260926 20:13:04.192341 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:13:04.194062 4715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368: No bootstrap required, opened a new log
I20260926 20:13:04.194881 4790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:04.195830 4790 log.cc:824] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:04.198189 4790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a: No bootstrap required, opened a new log
I20260926 20:13:04.199292 4866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:04.200273 4866 log.cc:824] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:04.202538 4790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } }
I20260926 20:13:04.202811 4790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:04.202870 4790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464c2847e5b84c7ea5d4c1ec8955928a, State: Initialized, Role: FOLLOWER
I20260926 20:13:04.203501 4790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } }
I20260926 20:13:04.203872 4715 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } }
I20260926 20:13:04.204159 4715 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:04.204259 4715 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd96985ec212447c9c28ff81f94f7368, State: Initialized, Role: FOLLOWER
I20260926 20:13:04.204797 4715 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } }
I20260926 20:13:04.205008 4790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:13:04.205402 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.195473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.23.50.190:43613)
0926 20:13:04.195585 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:04.195592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:04.195610 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:13:04.195654 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:04.195660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:04.195712 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:04.195799 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:04.196722 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.197456 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.198629 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.198938 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.199014 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.201231 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:04.201235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:04.201238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:04.201264 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:04.203430 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.203941 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.203945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.203946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.204035 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.204385 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.204389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.204390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.204616 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:04.204719 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:04.205094 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 20:13:04.205251 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:13:04.205514 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.195482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35482 (local address 127.23.50.188:32969)
0926 20:13:04.195599 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:04.195604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:04.196878 (+ 1274us) server_negotiation.cc:407] Connection header received
0926 20:13:04.197042 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:04.197046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:04.197093 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:04.197142 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:04.198084 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.198875 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.199979 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.200319 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.200399 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.202054 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:04.202058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:04.202061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:04.202087 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:04.203637 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.204117 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.204120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.204122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.204210 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.204528 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.204532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.204534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.204739 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:04.204848 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:04.205220 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 20:13:04.205350 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:13:04.206625 4715 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:13:04.211242 4897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:13:04.211320 4897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:13:04.211541 4883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } } }
I20260926 20:13:04.211679 4883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:13:04.209002 4722 master_runner.cc:186] Error getting master registration for 127.23.50.190:43613: OK, instance_id { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" instance_seqno: 1790453583694506 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:13:04.212253 4722 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:13:04.213868 4866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15: No bootstrap required, opened a new log
W20260926 20:13:04.215586 4906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 20:13:04.215636 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:04.196111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35720 (local address 127.23.50.189:37433)
0926 20:13:04.196328 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:04.196333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:04.206258 (+ 9925us) server_negotiation.cc:407] Connection header received
0926 20:13:04.206480 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:04.206484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:04.206545 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:04.206597 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:04.207708 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.208460 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.209719 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:04.210053 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:04.210131 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:04.211937 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:04.211941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:04.211944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:04.211972 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:04.213679 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.214241 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.214245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.214247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.214334 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:04.214646 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:04.214649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:04.214651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:04.214870 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:04.214970 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:04.215371 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 20:13:04.215489 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
W20260926 20:13:04.215668 4906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:13:04.216621 4884 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } } }
I20260926 20:13:04.216741 4884 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:13:04.215410 4647 master_runner.cc:186] Error getting master registration for 127.23.50.188:32969: OK, instance_id { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" instance_seqno: 1790453584140486 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:13:04.218961 4647 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:13:04.218847 4866 raft_consensus.cc:348] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } }
I20260926 20:13:04.219110 4866 raft_consensus.cc:374] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:04.219164 4866 raft_consensus.cc:729] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efaba2c9d4784a30bcb49673b834ce15, State: Initialized, Role: FOLLOWER
I20260926 20:13:04.220086 4866 consensus_queue.cc:260] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } }
I20260926 20:13:04.220679 4909 sys_catalog.cc:455] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } } }
I20260926 20:13:04.220834 4909 sys_catalog.cc:458] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:13:04.221680 4866 sys_catalog.cc:564] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:13:04.231736 4797 master_runner.cc:263] No leader master found from master efaba2c9d4784a30bcb49673b834ce15
I20260926 20:13:04.231843 4797 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 20:13:04.232692 4920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:13:04.232861 4920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:13:04.246233 4884 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:04.246493 4884 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } }
I20260926 20:13:04.247582 4884 leader_election.cc:288] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 464c2847e5b84c7ea5d4c1ec8955928a (127.23.50.189:37433), efaba2c9d4784a30bcb49673b834ce15 (127.23.50.188:32969)
I20260926 20:13:04.248458 4764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd96985ec212447c9c28ff81f94f7368" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" is_pre_election: true
I20260926 20:13:04.248807 4764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd96985ec212447c9c28ff81f94f7368 in term 0.
I20260926 20:13:04.249269 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd96985ec212447c9c28ff81f94f7368" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efaba2c9d4784a30bcb49673b834ce15" is_pre_election: true
I20260926 20:13:04.249606 4839 raft_consensus.cc:2454] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd96985ec212447c9c28ff81f94f7368 in term 0.
I20260926 20:13:04.250083 4663 leader_election.cc:302] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd96985ec212447c9c28ff81f94f7368, efaba2c9d4784a30bcb49673b834ce15; no voters:
I20260926 20:13:04.250749 4884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:13:04.250842 4884 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:13:04.250893 4884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:04.252241 4884 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } }
I20260926 20:13:04.252892 4764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd96985ec212447c9c28ff81f94f7368" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464c2847e5b84c7ea5d4c1ec8955928a"
I20260926 20:13:04.253023 4764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:04.254944 4764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd96985ec212447c9c28ff81f94f7368 in term 1.
I20260926 20:13:04.255234 4666 leader_election.cc:302] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 464c2847e5b84c7ea5d4c1ec8955928a, dd96985ec212447c9c28ff81f94f7368; no voters:
I20260926 20:13:04.255971 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dd96985ec212447c9c28ff81f94f7368" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efaba2c9d4784a30bcb49673b834ce15"
I20260926 20:13:04.256095 4839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:04.257360 4839 raft_consensus.cc:2454] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd96985ec212447c9c28ff81f94f7368 in term 1.
I20260926 20:13:04.257709 4884 leader_election.cc:288] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [CANDIDATE]: Term 1 election: Requested vote from peers 464c2847e5b84c7ea5d4c1ec8955928a (127.23.50.189:37433), efaba2c9d4784a30bcb49673b834ce15 (127.23.50.188:32969)
I20260926 20:13:04.257884 4884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:13:04.257979 4884 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [term 1 LEADER]: Becoming Leader. State: Replica: dd96985ec212447c9c28ff81f94f7368, State: Running, Role: LEADER
I20260926 20:13:04.258508 4884 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } }
I20260926 20:13:04.260569 4884 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd96985ec212447c9c28ff81f94f7368. Latest consensus state: current_term: 1 leader_uuid: "dd96985ec212447c9c28ff81f94f7368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } } }
I20260926 20:13:04.260689 4884 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:04.261087 4924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:13:04.262863 4924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:13:04.278136 4839 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [term 1 FOLLOWER]: Refusing update from remote peer dd96985ec212447c9c28ff81f94f7368: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:13:04.278873 4922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:13:04.286532 4909 sys_catalog.cc:455] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd96985ec212447c9c28ff81f94f7368. Latest consensus state: current_term: 1 leader_uuid: "dd96985ec212447c9c28ff81f94f7368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } } }
I20260926 20:13:04.286686 4909 sys_catalog.cc:458] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:13:04.288583 4764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [term 1 FOLLOWER]: Refusing update from remote peer dd96985ec212447c9c28ff81f94f7368: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:13:04.289223 4922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:13:04.293778 4883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd96985ec212447c9c28ff81f94f7368. Latest consensus state: current_term: 1 leader_uuid: "dd96985ec212447c9c28ff81f94f7368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } } }
I20260926 20:13:04.293943 4883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:13:04.301474 4922 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd96985ec212447c9c28ff81f94f7368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } } }
I20260926 20:13:04.301632 4922 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:04.303411 4883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd96985ec212447c9c28ff81f94f7368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } } }
I20260926 20:13:04.303550 4883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:13:04.304009 4925 mvcc.cc:205] Tried to move back new op lower bound from 7333697881194717184 to 7333697881126281216. Current Snapshot: MvccSnapshot[applied={T|T < 7333697881194717184}]
I20260926 20:13:04.306028 4928 mvcc.cc:205] Tried to move back new op lower bound from 7333697881194717184 to 7333697881126281216. Current Snapshot: MvccSnapshot[applied={T|T < 7333697881194717184}]
I20260926 20:13:04.306924 4884 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dd96985ec212447c9c28ff81f94f7368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } } }
I20260926 20:13:04.307088 4884 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368 [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:04.308300 4909 sys_catalog.cc:455] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dd96985ec212447c9c28ff81f94f7368" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd96985ec212447c9c28ff81f94f7368" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43613 } } peers { permanent_uuid: "464c2847e5b84c7ea5d4c1ec8955928a" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 37433 } } peers { permanent_uuid: "efaba2c9d4784a30bcb49673b834ce15" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 32969 } } }
I20260926 20:13:04.308432 4909 sys_catalog.cc:458] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:13:04.311340 4926 mvcc.cc:205] Tried to move back new op lower bound from 7333697881194717184 to 7333697881126281216. Current Snapshot: MvccSnapshot[applied={T|T < 7333697881194717184}]
I20260926 20:13:04.317919 4924 catalog_manager.cc:1347] Generated new cluster ID: 5dd6155fc0f04562b387c84ef01302ed
I20260926 20:13:04.318001 4924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:13:04.336675 4924 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:13:04.338152 4924 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:13:04.357329 4924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd96985ec212447c9c28ff81f94f7368: Generated new TSK 0
I20260926 20:13:04.360729 4924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:13:05.212611 4897 catalog_manager.cc:1259] Loaded cluster ID: 5dd6155fc0f04562b387c84ef01302ed
I20260926 20:13:05.212697 4897 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a: loading cluster ID for follower catalog manager: success
I20260926 20:13:05.214363 4897 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a: acquiring CA information for follower catalog manager: success
I20260926 20:13:05.215512 4897 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 464c2847e5b84c7ea5d4c1ec8955928a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:13:05.234150 4920 catalog_manager.cc:1259] Loaded cluster ID: 5dd6155fc0f04562b387c84ef01302ed
I20260926 20:13:05.234227 4920 catalog_manager.cc:1552] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15: loading cluster ID for follower catalog manager: success
I20260926 20:13:05.235879 4920 catalog_manager.cc:1574] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15: acquiring CA information for follower catalog manager: success
I20260926 20:13:05.237052 4920 catalog_manager.cc:1602] T 00000000000000000000000000000000 P efaba2c9d4784a30bcb49673b834ce15: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:13:05.436360 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 4647
I20260926 20:13:05.442628 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 4722
I20260926 20:13:05.448246 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 4797
2026-09-26T20:13:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3144 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:13:05 dist-test-slave-7034 krb5kdc[4942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:13:05 dist-test-slave-7034 krb5kdc[4942](info): set up 2 sockets
Sep 26 20:13:05 dist-test-slave-7034 krb5kdc[4942](info): commencing operation
krb5kdc: starting...
W20260926 20:13:06.486559 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:13:06 dist-test-slave-7034 krb5kdc[4942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:13:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:13:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:13:06.612360 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42553
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:42553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:06.726051 4955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:06.726369 4955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:06.726451 4955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:06.730604 4955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:13:06.730691 4955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:06.730727 4955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:06.730768 4955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:13:06.730808 4955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:13:06.736158 4955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:42553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:06.737329 4955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:06.738461 4955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:06.747244 4961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:06.747272 4960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:06.747395 4963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:06.747642 4955 server_base.cc:1034] running on GCE node
I20260926 20:13:06.748358 4955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:06.749667 4955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:06.750888 4955 hybrid_clock.cc:648] HybridClock initialized: now 1790453586750880 us; error 35 us; skew 500 ppm
Sep 26 20:13:06 dist-test-slave-7034 krb5kdc[4942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453586, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:06.754120 4955 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:13:06.755296 4955 webserver.cc:460] Webserver started at http://127.23.50.190:37097/ using document root <none> and password file <none>
I20260926 20:13:06.755932 4955 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:06.756036 4955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:06.756307 4955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:13:06.758701 4955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "1bd83ca4bd6c4259b1dd6202a32e0037"
format_stamp: "Formatted at 2026-09-26 20:13:06 on dist-test-slave-7034"
server_key: "23a2b25c8f1d01398eb7146c80e89a6c"
server_key_iv: "34371751e9f9cdc9ff54e6152e5ad596"
server_key_version: "encryptionkey@0"
I20260926 20:13:06.759364 4955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "1bd83ca4bd6c4259b1dd6202a32e0037"
format_stamp: "Formatted at 2026-09-26 20:13:06 on dist-test-slave-7034"
server_key: "23a2b25c8f1d01398eb7146c80e89a6c"
server_key_iv: "34371751e9f9cdc9ff54e6152e5ad596"
server_key_version: "encryptionkey@0"
I20260926 20:13:06.763499 4955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:13:06.766332 4970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:06.767724 4955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:13:06.767884 4955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "1bd83ca4bd6c4259b1dd6202a32e0037"
format_stamp: "Formatted at 2026-09-26 20:13:06 on dist-test-slave-7034"
server_key: "23a2b25c8f1d01398eb7146c80e89a6c"
server_key_iv: "34371751e9f9cdc9ff54e6152e5ad596"
server_key_version: "encryptionkey@0"
I20260926 20:13:06.768018 4955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:06.794308 4955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:06.797375 4955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:06.797601 4955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:06.805738 4955 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:42553
I20260926 20:13:06.805775 5022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:42553 every 8 connection(s)
I20260926 20:13:06.806805 4955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:13:06.808902 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 4955
I20260926 20:13:06.809042 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:13:06.809365 23754 external_mini_cluster.cc:1428] Setting key 09889876a5372b13a49d3e46aac2b046
I20260926 20:13:06.809973 5023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:06.816491 5023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037: Bootstrap starting.
Sep 26 20:13:06 dist-test-slave-7034 krb5kdc[4942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:06.819366 5023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:06.820348 5023 log.cc:824] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:06.822557 5023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037: No bootstrap required, opened a new log
I20260926 20:13:06.824421 5026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:06.810759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.23.50.190:42553)
0926 20:13:06.811232 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:06.811254 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:06.811297 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:13:06.812013 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:06.812030 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:06.812382 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:06.812789 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:06.813748 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:06.814834 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:06.815750 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:06.816233 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:06.816400 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:06.819252 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:06.819265 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:06.819279 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:06.819312 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:06.821378 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:06.821876 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:06.821885 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:06.821891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:06.822014 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:06.822317 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:06.822321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:06.822322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:06.822877 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:06.823083 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:06.823385 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 20:13:06.823694 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":152,"threads_started":1}
I20260926 20:13:06.826854 5023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd83ca4bd6c4259b1dd6202a32e0037" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42553 } }
I20260926 20:13:06.827164 5023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:06.827224 5023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bd83ca4bd6c4259b1dd6202a32e0037, State: Initialized, Role: FOLLOWER
I20260926 20:13:06.827553 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:13:06.827683 5023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd83ca4bd6c4259b1dd6202a32e0037" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42553 } }
I20260926 20:13:06.827845 5023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:13:06.827910 5023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:13:06.828008 5023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:06.829214 5023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd83ca4bd6c4259b1dd6202a32e0037" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42553 } }
I20260926 20:13:06.829589 5023 leader_election.cc:302] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bd83ca4bd6c4259b1dd6202a32e0037; no voters:
I20260926 20:13:06.829948 5023 leader_election.cc:288] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:13:06.830072 5028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:13:06.830327 5028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [term 1 LEADER]: Becoming Leader. State: Replica: 1bd83ca4bd6c4259b1dd6202a32e0037, State: Running, Role: LEADER
I20260926 20:13:06.830670 5028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd83ca4bd6c4259b1dd6202a32e0037" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42553 } }
I20260926 20:13:06.831208 5023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:13:06.832288 5030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bd83ca4bd6c4259b1dd6202a32e0037. Latest consensus state: current_term: 1 leader_uuid: "1bd83ca4bd6c4259b1dd6202a32e0037" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd83ca4bd6c4259b1dd6202a32e0037" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42553 } } }
I20260926 20:13:06.832430 5030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:06.832849 5037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:13:06.832796 5029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bd83ca4bd6c4259b1dd6202a32e0037" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bd83ca4bd6c4259b1dd6202a32e0037" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42553 } } }
I20260926 20:13:06.833567 5029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037 [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:06.836076 5037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:13:06.841679 5037 catalog_manager.cc:1347] Generated new cluster ID: 8e1edbb3e2d6459b9a44e1ab28608386
I20260926 20:13:06.841784 5037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:13:06.874316 5037 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:13:06.875303 5037 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:13:06.880060 5037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bd83ca4bd6c4259b1dd6202a32e0037: Generated new TSK 0
I20260926 20:13:06.880924 5037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:13:06.935058 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 4955
2026-09-26T20:13:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1486 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:13:06 dist-test-slave-7034 krb5kdc[5050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:13:06 dist-test-slave-7034 krb5kdc[5050](info): set up 2 sockets
Sep 26 20:13:06 dist-test-slave-7034 krb5kdc[5050](info): commencing operation
krb5kdc: starting...
W20260926 20:13:07.974694 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:13:08 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:13:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:13:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:13:08.099169 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37837
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:37837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:08.210355 5063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:08.210632 5063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:08.210673 5063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:08.214641 5063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:13:08.214699 5063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:08.214718 5063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:08.214735 5063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:13:08.214749 5063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:13:08.220209 5063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:37837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:08.221251 5063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:08.222294 5063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:08.230922 5068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:08.230922 5071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:08.231112 5063 server_base.cc:1034] running on GCE node
W20260926 20:13:08.230929 5069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:08.231853 5063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:08.233038 5063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:08.234244 5063 hybrid_clock.cc:648] HybridClock initialized: now 1790453588234206 us; error 49 us; skew 500 ppm
Sep 26 20:13:08 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:08.237149 5063 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:13:08.238229 5063 webserver.cc:460] Webserver started at http://127.23.50.190:38029/ using document root <none> and password file <none>
I20260926 20:13:08.238821 5063 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:08.238914 5063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:08.239151 5063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:13:08.241225 5063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "a8e2fc54aee34b0491de18484326bdbf"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "12e14180ece8b02ba4ad686d700f274b"
server_key_iv: "aaf4a5b3146098f89f5dfc7fd02c3784"
server_key_version: "encryptionkey@0"
I20260926 20:13:08.241808 5063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "a8e2fc54aee34b0491de18484326bdbf"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "12e14180ece8b02ba4ad686d700f274b"
server_key_iv: "aaf4a5b3146098f89f5dfc7fd02c3784"
server_key_version: "encryptionkey@0"
I20260926 20:13:08.245510 5063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:13:08.247960 5078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:08.249301 5063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:13:08.249449 5063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "a8e2fc54aee34b0491de18484326bdbf"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "12e14180ece8b02ba4ad686d700f274b"
server_key_iv: "aaf4a5b3146098f89f5dfc7fd02c3784"
server_key_version: "encryptionkey@0"
I20260926 20:13:08.249569 5063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:08.264300 5063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:08.267575 5063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:08.267794 5063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:08.274935 5063 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:37837
I20260926 20:13:08.274961 5130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:37837 every 8 connection(s)
I20260926 20:13:08.275909 5063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:13:08.278644 5131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:08.283969 5131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Bootstrap starting.
I20260926 20:13:08.285372 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 5063
I20260926 20:13:08.285511 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:13:08.285784 23754 external_mini_cluster.cc:1428] Setting key 38cb6baac6c29a018e8742475a250d61
I20260926 20:13:08.286700 5131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:08.287621 5131 log.cc:824] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:08.289816 5131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: No bootstrap required, opened a new log
I20260926 20:13:08.294272 5131 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } }
I20260926 20:13:08.294529 5131 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:08.294626 5131 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8e2fc54aee34b0491de18484326bdbf, State: Initialized, Role: FOLLOWER
I20260926 20:13:08.295168 5131 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } }
Sep 26 20:13:08 dist-test-slave-7034 krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:08.295326 5131 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:13:08.295392 5131 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:13:08.295492 5131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:08.296768 5131 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } }
I20260926 20:13:08.297186 5131 leader_election.cc:302] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8e2fc54aee34b0491de18484326bdbf; no voters:
I20260926 20:13:08.297573 5131 leader_election.cc:288] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:13:08.297667 5136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:13:08.297950 5136 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 1 LEADER]: Becoming Leader. State: Replica: a8e2fc54aee34b0491de18484326bdbf, State: Running, Role: LEADER
I20260926 20:13:08.298329 5136 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } }
I20260926 20:13:08.299031 5131 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:13:08.300032 5137 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8e2fc54aee34b0491de18484326bdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } } }
I20260926 20:13:08.300156 5137 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:08.300529 5138 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8e2fc54aee34b0491de18484326bdbf. Latest consensus state: current_term: 1 leader_uuid: "a8e2fc54aee34b0491de18484326bdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } } }
I20260926 20:13:08.300681 5138 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:08.301062 5145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:13:08.302073 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:08.286902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.23.50.190:37837)
0926 20:13:08.287395 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:08.287417 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:08.287461 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:13:08.288280 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:08.288300 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:08.288745 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:08.289111 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:08.289812 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:08.291335 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:08.292191 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:08.293025 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:08.293191 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:08.295963 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:08.295969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:08.295986 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:08.296019 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:08.298077 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:08.298609 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:08.298614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:08.298621 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:08.298767 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:08.299116 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:08.299120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:08.299122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:08.299660 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:08.300550 (+ 890us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:08.300729 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:13:08.301048 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":161,"threads_started":1}
I20260926 20:13:08.304260 5145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:13:08.309896 5145 catalog_manager.cc:1347] Generated new cluster ID: e490f6a2b7dd4274b8d468d0ebe75d9e
I20260926 20:13:08.309960 5145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:13:08.324630 5145 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:13:08.326028 5145 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:13:08.332899 5145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Generated new TSK 0
I20260926 20:13:08.333706 5145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:13:08.368928 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37837
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:08.481125 5157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:08.481411 5157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:08.481487 5157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:08.485498 5157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:08.485589 5157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:08.485682 5157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:13:08.490963 5157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:08.492026 5157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:08.493031 5157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:08.503139 5163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:08.503194 5162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:08.503163 5157 server_base.cc:1034] running on GCE node
W20260926 20:13:08.503348 5165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:08.503813 5157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:08.504472 5157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:08.505645 5157 hybrid_clock.cc:648] HybridClock initialized: now 1790453588505607 us; error 48 us; skew 500 ppm
Sep 26 20:13:08 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:08.508558 5157 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:13:08.509477 5157 webserver.cc:460] Webserver started at http://127.23.50.129:44147/ using document root <none> and password file <none>
I20260926 20:13:08.510118 5157 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:08.510210 5157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:08.510444 5157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:13:08.512466 5157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "82531edff95844979c4610ec2b514f66"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "255214cff6f7a25daa59f2b4d911e312"
server_key_iv: "ca4b0b17050971bd73c27494019c8fc0"
server_key_version: "encryptionkey@0"
I20260926 20:13:08.513032 5157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "82531edff95844979c4610ec2b514f66"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "255214cff6f7a25daa59f2b4d911e312"
server_key_iv: "ca4b0b17050971bd73c27494019c8fc0"
server_key_version: "encryptionkey@0"
I20260926 20:13:08.516695 5157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 20:13:08.519129 5172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:08.520385 5157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:13:08.520500 5157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "82531edff95844979c4610ec2b514f66"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "255214cff6f7a25daa59f2b4d911e312"
server_key_iv: "ca4b0b17050971bd73c27494019c8fc0"
server_key_version: "encryptionkey@0"
I20260926 20:13:08.520601 5157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:08.537976 5157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:08.541345 5157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:08.541560 5157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:08.542202 5157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:08.543045 5157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:13:08.543092 5157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:08.543165 5157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:08.543196 5157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:08.552517 5157 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:41785
I20260926 20:13:08.552561 5285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:41785 every 8 connection(s)
I20260926 20:13:08.553475 5157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:13:08.555896 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 5157
I20260926 20:13:08.556037 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:13:08.556389 23754 external_mini_cluster.cc:1428] Setting key 0f783ee5dcdd88778073d89ef33bc938
Sep 26 20:13:08 dist-test-slave-7034 krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
I20260926 20:13:08.569150 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:08.555465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:37035 (local address 127.23.50.190:37837)
0926 20:13:08.555615 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:08.555620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:08.556711 (+ 1091us) server_negotiation.cc:407] Connection header received
0926 20:13:08.558322 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:08.558327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:08.558390 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:08.558483 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:08.559815 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:08.560341 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:08.562193 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:08.562423 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:08.562506 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:08.565416 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:08.565420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:08.565422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:08.565446 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:08.567009 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:08.567546 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:08.567549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:08.567551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:08.567634 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:08.567982 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:08.567987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:08.567988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:08.568194 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:08.568303 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:08.568861 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 20:13:08.569007 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:13:08.570307 5288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:08.555962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:37035)
0926 20:13:08.556539 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:08.556598 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:08.558115 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:08.558604 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:08.558611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:08.558968 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:08.559648 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:08.559669 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:08.560470 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:08.560476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:08.562059 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:08.562075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:08.562584 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:08.562591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:08.562841 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:08.563495 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:08.563523 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:08.565239 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:08.567152 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:08.567158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:08.567173 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:08.567437 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:08.567732 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:08.567736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:08.567738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:08.567885 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:08.568302 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:08.568308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:08.568558 (+ 250us) client_negotiation.cc:769] Sending connection context
0926 20:13:08.568763 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 20:13:08.569098 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":142,"threads_started":1}
I20260926 20:13:08.571624 5286 heartbeater.cc:347] Connected to a master server at 127.23.50.190:37837
I20260926 20:13:08.571892 5286 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:08.572378 5286 heartbeater.cc:511] Master 127.23.50.190:37837 requested a full tablet report, sending...
I20260926 20:13:08.573966 5095 ts_manager.cc:194] Registered new tserver with Master: 82531edff95844979c4610ec2b514f66 (127.23.50.129:41785)
I20260926 20:13:08.575162 5095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:37035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:13:08.587271 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37837
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:08.698908 5291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:08.699270 5291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:08.699365 5291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:08.703676 5291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:08.703766 5291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:08.703864 5291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:13:08.709282 5291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:08.710489 5291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:08.711545 5291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:08.721539 5297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:08.721771 5291 server_base.cc:1034] running on GCE node
W20260926 20:13:08.721539 5296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:08.721567 5299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:08.722361 5291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:08.723043 5291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:08.724241 5291 hybrid_clock.cc:648] HybridClock initialized: now 1790453588724208 us; error 48 us; skew 500 ppm
Sep 26 20:13:08 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:08.732395 5291 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:13:08.733412 5291 webserver.cc:460] Webserver started at http://127.23.50.130:44513/ using document root <none> and password file <none>
I20260926 20:13:08.734133 5291 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:08.734232 5291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:08.734479 5291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:13:08.736656 5291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "e1a6c17d045c429e9609fdf788430a59"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "d4e3d005143a2995d852e1d419b4ad54"
server_key_iv: "fe5e4bcf357100fca99769afc007b6f5"
server_key_version: "encryptionkey@0"
I20260926 20:13:08.737269 5291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "e1a6c17d045c429e9609fdf788430a59"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "d4e3d005143a2995d852e1d419b4ad54"
server_key_iv: "fe5e4bcf357100fca99769afc007b6f5"
server_key_version: "encryptionkey@0"
I20260926 20:13:08.741384 5291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:13:08.744194 5306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:08.745575 5291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:13:08.745730 5291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "e1a6c17d045c429e9609fdf788430a59"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "d4e3d005143a2995d852e1d419b4ad54"
server_key_iv: "fe5e4bcf357100fca99769afc007b6f5"
server_key_version: "encryptionkey@0"
I20260926 20:13:08.745898 5291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:08.774336 5291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:08.777967 5291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:08.778177 5291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:08.778812 5291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:08.779680 5291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:13:08.779749 5291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:08.779846 5291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:08.779901 5291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:08.788767 5291 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:37309
I20260926 20:13:08.788808 5419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:37309 every 8 connection(s)
I20260926 20:13:08.789680 5291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:13:08.794862 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 5291
I20260926 20:13:08.795012 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:13:08.795313 23754 external_mini_cluster.cc:1428] Setting key fec9fa2f3e1003bff278cbfe339e877e
Sep 26 20:13:08 dist-test-slave-7034 krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:08.805655 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:08.791612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:59669 (local address 127.23.50.190:37837)
0926 20:13:08.791759 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:08.791762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:08.792514 (+ 752us) server_negotiation.cc:407] Connection header received
0926 20:13:08.794005 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:08.794009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:08.794068 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:08.794182 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:08.795723 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:08.796436 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:08.798346 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:08.798613 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:08.798734 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:08.801924 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:08.801927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:08.801929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:08.801952 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:08.803412 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:08.804039 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:08.804041 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:08.804043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:08.804114 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:08.804512 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:08.804515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:08.804516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:08.804683 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:08.804758 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:08.805443 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 20:13:08.805535 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:13:08.806764 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:08.791904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:59669)
0926 20:13:08.792357 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:08.792402 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:08.793738 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:08.794359 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:08.794366 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:08.794853 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:08.795545 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:08.795567 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:08.796711 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:08.796715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:08.798222 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:08.798237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:08.798787 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:08.798794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:08.798968 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:08.799618 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:08.799640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:08.801700 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:08.803537 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:08.803546 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:08.803566 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:08.803937 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:08.804212 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:08.804218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:08.804221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:08.804419 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:08.804794 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:08.804801 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:08.805161 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 20:13:08.805402 (+ 241us) client_negotiation.cc:240] Negotiation successful
0926 20:13:08.805755 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260926 20:13:08.808015 5420 heartbeater.cc:347] Connected to a master server at 127.23.50.190:37837
I20260926 20:13:08.808298 5420 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:08.808820 5420 heartbeater.cc:511] Master 127.23.50.190:37837 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
I20260926 20:13:08.810046 5095 ts_manager.cc:194] Registered new tserver with Master: e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309)
I20260926 20:13:08.810815 5095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:59669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:13:08.825166 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37837
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:08.936206 5425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:08.936494 5425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:08.936539 5425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:08.940625 5425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:08.940693 5425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:08.940753 5425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:13:08.946039 5425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:08.947090 5425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:08.948068 5425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:08.957798 5431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:08.957806 5433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:08.957937 5430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:08.958109 5425 server_base.cc:1034] running on GCE node
I20260926 20:13:08.958570 5425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:08.959183 5425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:08.960368 5425 hybrid_clock.cc:648] HybridClock initialized: now 1790453588960353 us; error 28 us; skew 500 ppm
Sep 26 20:13:08 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:08.963238 5425 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:13:08.964180 5425 webserver.cc:460] Webserver started at http://127.23.50.131:38689/ using document root <none> and password file <none>
I20260926 20:13:08.964753 5425 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:08.964850 5425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:08.965083 5425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:13:08.967140 5425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "0b5d71e02915496594f99d7cc851d253"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "a98a1f65a734d29eb6c7dcd74d207972"
server_key_iv: "8acbc1feaeffe06bf5426b142205254f"
server_key_version: "encryptionkey@0"
I20260926 20:13:08.967722 5425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "0b5d71e02915496594f99d7cc851d253"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "a98a1f65a734d29eb6c7dcd74d207972"
server_key_iv: "8acbc1feaeffe06bf5426b142205254f"
server_key_version: "encryptionkey@0"
I20260926 20:13:08.971585 5425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:13:08.974148 5440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:08.975356 5425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:13:08.975493 5425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "0b5d71e02915496594f99d7cc851d253"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "a98a1f65a734d29eb6c7dcd74d207972"
server_key_iv: "8acbc1feaeffe06bf5426b142205254f"
server_key_version: "encryptionkey@0"
I20260926 20:13:08.975605 5425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:08.990903 5425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:08.994287 5425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:08.994493 5425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:08.995085 5425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:08.995900 5425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:13:08.995970 5425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:08.996043 5425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:08.996079 5425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:09.005748 5425 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:36255
I20260926 20:13:09.005769 5553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:36255 every 8 connection(s)
I20260926 20:13:09.006675 5425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:13:09.011620 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 5425
I20260926 20:13:09.011755 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:13:09.012045 23754 external_mini_cluster.cc:1428] Setting key 83a0354f8d1ef8b49cedf6fd670a5358
Sep 26 20:13:09 dist-test-slave-7034 krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453588, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:09.020704 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.008543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:59543 (local address 127.23.50.190:37837)
0926 20:13:09.008694 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:09.008698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:09.009585 (+ 887us) server_negotiation.cc:407] Connection header received
0926 20:13:09.011030 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:09.011033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:09.011094 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:09.011172 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:09.012491 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.013032 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.014345 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.014575 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.014686 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.017021 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:09.017025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:09.017027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:09.017061 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:09.018624 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:09.019175 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:09.019179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:09.019181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:09.019261 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:09.019600 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:09.019605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:09.019607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:09.019810 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:09.019918 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:09.020410 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 20:13:09.020531 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:13:09.021513 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.008972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:59543)
0926 20:13:09.009422 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:09.009465 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:09.010811 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:09.011344 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:09.011351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:09.011762 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:09.012338 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.012354 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.013160 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.013163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.014217 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.014227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.014721 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.014725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.014903 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.015429 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:09.015445 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:09.016865 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:09.018748 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:09.018754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:09.018769 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:09.019069 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:09.019365 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:09.019368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:09.019370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:09.019505 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:09.019915 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:09.019922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:09.020169 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 20:13:09.020371 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 20:13:09.020692 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
I20260926 20:13:09.022645 5554 heartbeater.cc:347] Connected to a master server at 127.23.50.190:37837
I20260926 20:13:09.022899 5554 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:09.023406 5554 heartbeater.cc:511] Master 127.23.50.190:37837 requested a full tablet report, sending...
I20260926 20:13:09.024456 5095 ts_manager.cc:194] Registered new tserver with Master: 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255)
I20260926 20:13:09.025089 5095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:59543
I20260926 20:13:09.025933 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:13:09.035418 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.027793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.23.50.190:37837)
0926 20:13:09.027915 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:09.027919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:09.028115 (+ 196us) server_negotiation.cc:407] Connection header received
0926 20:13:09.028250 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:09.028253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:09.028298 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:09.028384 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:09.029079 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.029576 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.030477 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.030696 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.030813 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.032046 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:09.032050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:09.032052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:09.032072 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:09.033157 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:09.033638 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:09.033641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:09.033642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:09.033751 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:09.034060 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:09.034064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:09.034066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:09.034350 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:09.034479 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:09.035155 (+ 676us) server_negotiation.cc:299] Negotiation successful
0926 20:13:09.035268 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:13:09.042994 5095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:13:09.046003 5095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:13:09.062975 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.056893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.23.50.129:41785)
0926 20:13:09.057708 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:09.057724 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:09.057748 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:13:09.057822 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:09.057827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:09.058055 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:09.058195 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:09.058599 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.059693 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.061388 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.062253 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.062385 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.062548 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:09.062699 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 20:13:09.062795 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":107,"threads_started":1}
I20260926 20:13:09.062960 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.056598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:41785 (local address 127.0.0.1:51300)
0926 20:13:09.056847 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:09.056860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:09.056942 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:09.058361 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:09.058364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:09.058382 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:09.058481 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.058487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.059841 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.059845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.061234 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.061246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.062384 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.062387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.062505 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.062544 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 20:13:09.062667 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 20:13:09.062795 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
I20260926 20:13:09.063653 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.055857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:37309 (local address 127.0.0.1:40904)
0926 20:13:09.056283 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:09.056343 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:09.056501 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:09.057211 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:09.057232 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:09.057260 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:09.057399 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.057407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.060737 (+ 3330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.060743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.061804 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.061814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.063156 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.063160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.063312 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.063372 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 20:13:09.063447 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:13:09.063521 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":98,"thread_start_us":90,"threads_started":1}
I20260926 20:13:09.063766 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.055344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.23.50.130:37309)
0926 20:13:09.056068 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:09.056081 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:09.056465 (+ 384us) server_negotiation.cc:407] Connection header received
0926 20:13:09.056767 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:09.056773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:09.057016 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:09.057199 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:09.057536 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.058703 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.061954 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.062820 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.062944 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.063091 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:09.063504 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 20:13:09.063613 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":190,"threads_started":1}
I20260926 20:13:09.063757 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.055340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.23.50.131:36255)
0926 20:13:09.055805 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:09.055826 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:09.058772 (+ 2946us) server_negotiation.cc:407] Connection header received
0926 20:13:09.058906 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:09.058912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:09.059085 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:09.059230 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:09.059457 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.060555 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.062122 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.062892 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.062981 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.063091 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:09.063507 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 20:13:09.063608 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":178,"threads_started":1}
I20260926 20:13:09.064405 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.057251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:36255 (local address 127.0.0.1:50678)
0926 20:13:09.058671 (+ 1420us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:09.058684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:09.058793 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:09.059234 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:09.059238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:09.059256 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:09.059358 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.059364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.060765 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.060769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.061993 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.062008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.063156 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.063160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.063303 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.063368 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 20:13:09.064199 (+ 831us) client_negotiation.cc:240] Negotiation successful
0926 20:13:09.064272 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1350,"mutex_wait_us":73,"thread_start_us":90,"threads_started":1}
I20260926 20:13:09.066915 5354 tablet_service.cc:1467] Processing CreateTablet for tablet cc8798329dc44bdaab4542e15b07b96c (DEFAULT_TABLE table=test-table [id=e201128c2a4b42c4a2e9cf48fe69da20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:09.066802 5220 tablet_service.cc:1467] Processing CreateTablet for tablet cc8798329dc44bdaab4542e15b07b96c (DEFAULT_TABLE table=test-table [id=e201128c2a4b42c4a2e9cf48fe69da20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:09.067934 5220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8798329dc44bdaab4542e15b07b96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:09.067940 5354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8798329dc44bdaab4542e15b07b96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:09.067759 5488 tablet_service.cc:1467] Processing CreateTablet for tablet cc8798329dc44bdaab4542e15b07b96c (DEFAULT_TABLE table=test-table [id=e201128c2a4b42c4a2e9cf48fe69da20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:09.068703 5488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc8798329dc44bdaab4542e15b07b96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:09.074113 5571 tablet_bootstrap.cc:492] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: Bootstrap starting.
I20260926 20:13:09.075510 5572 tablet_bootstrap.cc:492] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Bootstrap starting.
I20260926 20:13:09.075870 5573 tablet_bootstrap.cc:492] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253: Bootstrap starting.
I20260926 20:13:09.076679 5571 tablet_bootstrap.cc:654] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:09.077422 5571 log.cc:824] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:09.077731 5572 tablet_bootstrap.cc:654] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:09.077731 5573 tablet_bootstrap.cc:654] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:09.078747 5572 log.cc:824] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:09.079011 5573 log.cc:824] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:09.079794 5571 tablet_bootstrap.cc:492] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: No bootstrap required, opened a new log
I20260926 20:13:09.079943 5571 ts_tablet_manager.cc:1397] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:13:09.080649 5572 tablet_bootstrap.cc:492] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: No bootstrap required, opened a new log
I20260926 20:13:09.080881 5572 ts_tablet_manager.cc:1397] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:13:09.082952 5571 raft_consensus.cc:348] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:09.083184 5571 raft_consensus.cc:374] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:09.083252 5571 raft_consensus.cc:729] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a6c17d045c429e9609fdf788430a59, State: Initialized, Role: FOLLOWER
I20260926 20:13:09.083464 5573 tablet_bootstrap.cc:492] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253: No bootstrap required, opened a new log
I20260926 20:13:09.083720 5573 ts_tablet_manager.cc:1397] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 20:13:09.083737 5571 consensus_queue.cc:260] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:09.084420 5420 heartbeater.cc:503] Master 127.23.50.190:37837 was elected leader, sending a full tablet report...
I20260926 20:13:09.084895 5572 raft_consensus.cc:348] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:09.085136 5572 raft_consensus.cc:374] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:09.085188 5572 raft_consensus.cc:729] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82531edff95844979c4610ec2b514f66, State: Initialized, Role: FOLLOWER
I20260926 20:13:09.085619 5572 consensus_queue.cc:260] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:09.086308 5286 heartbeater.cc:503] Master 127.23.50.190:37837 was elected leader, sending a full tablet report...
I20260926 20:13:09.086637 5572 ts_tablet_manager.cc:1428] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260926 20:13:09.087543 5571 ts_tablet_manager.cc:1428] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260926 20:13:09.087477 5573 raft_consensus.cc:348] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:09.087800 5573 raft_consensus.cc:374] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:09.087893 5573 raft_consensus.cc:729] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5d71e02915496594f99d7cc851d253, State: Initialized, Role: FOLLOWER
I20260926 20:13:09.088344 5573 consensus_queue.cc:260] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:09.089011 5554 heartbeater.cc:503] Master 127.23.50.190:37837 was elected leader, sending a full tablet report...
I20260926 20:13:09.089257 5573 ts_tablet_manager.cc:1428] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 20:13:09.257710 5555 tablet.cc:2369] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:13:09.270931 5577 raft_consensus.cc:482] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:09.271191 5577 raft_consensus.cc:504] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:09.272368 5577 leader_election.cc:288] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255), 82531edff95844979c4610ec2b514f66 (127.23.50.129:41785)
I20260926 20:13:09.276814 5422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.272656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:41785 (local address 127.23.50.130:34491)
0926 20:13:09.272878 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:09.272895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:09.272992 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:09.273362 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:09.273365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:09.273406 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:09.273688 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.273693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.274805 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.274807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.275561 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.275569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.276403 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.276406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.276506 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.276530 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:13:09.276598 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 20:13:09.276666 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":110,"spinlock_wait_cycles":5888}
I20260926 20:13:09.276847 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.272776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:34491 (local address 127.23.50.129:41785)
0926 20:13:09.272911 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:09.272916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:09.272939 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:13:09.273137 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:09.273141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:09.273196 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:09.273326 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:09.273809 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.274665 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.275693 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.276283 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.276402 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.276456 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:09.276632 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 20:13:09.276722 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:13:09.277242 5579 raft_consensus.cc:482] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:09.277516 5580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.272656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:36255 (local address 127.23.50.130:59861)
0926 20:13:09.273005 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:09.273018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:09.273115 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:09.273362 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:09.273365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:09.273404 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:09.273686 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.273693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.274814 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.274818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.275947 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.275958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.276858 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.276865 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.276973 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.277202 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 20:13:09.277291 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:13:09.277374 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":94,"spinlock_wait_cycles":29440,"thread_start_us":92,"threads_started":1}
I20260926 20:13:09.277515 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.272791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:59861 (local address 127.23.50.131:36255)
0926 20:13:09.272911 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:09.272916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:09.273100 (+ 184us) server_negotiation.cc:407] Connection header received
0926 20:13:09.273182 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:09.273186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:09.273235 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:09.273324 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:09.273809 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.274625 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.276068 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.276639 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.276755 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.276798 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:09.277326 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 20:13:09.277405 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:13:09.277493 5579 raft_consensus.cc:504] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:09.278260 5508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8798329dc44bdaab4542e15b07b96c" candidate_uuid: "e1a6c17d045c429e9609fdf788430a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5d71e02915496594f99d7cc851d253" is_pre_election: true
I20260926 20:13:09.278471 5508 raft_consensus.cc:2454] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a6c17d045c429e9609fdf788430a59 in term 0.
I20260926 20:13:09.278944 5579 leader_election.cc:288] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309), 82531edff95844979c4610ec2b514f66 (127.23.50.129:41785)
I20260926 20:13:09.278903 5307 leader_election.cc:302] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5d71e02915496594f99d7cc851d253, e1a6c17d045c429e9609fdf788430a59; no voters:
I20260926 20:13:09.279166 5577 raft_consensus.cc:2781] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:13:09.279256 5577 raft_consensus.cc:482] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:13:09.279304 5577 raft_consensus.cc:3037] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:09.280612 5577 raft_consensus.cc:504] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:09.280917 5577 leader_election.cc:288] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [CANDIDATE]: Term 1 election: Requested vote from peers 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255), 82531edff95844979c4610ec2b514f66 (127.23.50.129:41785)
I20260926 20:13:09.281217 5508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8798329dc44bdaab4542e15b07b96c" candidate_uuid: "e1a6c17d045c429e9609fdf788430a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5d71e02915496594f99d7cc851d253"
I20260926 20:13:09.281348 5508 raft_consensus.cc:3037] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:09.282572 5240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8798329dc44bdaab4542e15b07b96c" candidate_uuid: "e1a6c17d045c429e9609fdf788430a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82531edff95844979c4610ec2b514f66" is_pre_election: true
I20260926 20:13:09.282637 5508 raft_consensus.cc:2454] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a6c17d045c429e9609fdf788430a59 in term 1.
I20260926 20:13:09.282838 5240 raft_consensus.cc:2454] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1a6c17d045c429e9609fdf788430a59 in term 0.
I20260926 20:13:09.282920 5307 leader_election.cc:302] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5d71e02915496594f99d7cc851d253, e1a6c17d045c429e9609fdf788430a59; no voters:
I20260926 20:13:09.283069 5577 raft_consensus.cc:2781] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:13:09.283128 5239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8798329dc44bdaab4542e15b07b96c" candidate_uuid: "e1a6c17d045c429e9609fdf788430a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82531edff95844979c4610ec2b514f66"
I20260926 20:13:09.283252 5577 raft_consensus.cc:686] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 1 LEADER]: Becoming Leader. State: Replica: e1a6c17d045c429e9609fdf788430a59, State: Running, Role: LEADER
I20260926 20:13:09.283242 5239 raft_consensus.cc:3037] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:09.283572 5577 consensus_queue.cc:237] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:09.283653 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.279114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:37309 (local address 127.23.50.131:59273)
0926 20:13:09.279635 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:09.279649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:09.279746 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:09.280063 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:09.280066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:09.280083 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:09.280175 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.280180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.281126 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.281129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.282343 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.282354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.283239 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.283243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.283358 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.283383 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:13:09.283459 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:13:09.283538 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463}
I20260926 20:13:09.284068 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.279195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:59273 (local address 127.23.50.130:37309)
0926 20:13:09.279302 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:09.279307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:09.279726 (+ 419us) server_negotiation.cc:407] Connection header received
0926 20:13:09.279844 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:09.279847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:09.279891 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:09.279966 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:09.280269 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.281000 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.282477 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.283017 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.283127 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.283177 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:09.283896 (+ 719us) server_negotiation.cc:299] Negotiation successful
0926 20:13:09.283967 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 20:13:09.284561 5239 raft_consensus.cc:2454] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1a6c17d045c429e9609fdf788430a59 in term 1.
I20260926 20:13:09.286103 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.279960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:53721 (local address 127.23.50.129:41785)
0926 20:13:09.280158 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:09.280161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:09.282110 (+ 1949us) server_negotiation.cc:407] Connection header received
0926 20:13:09.282288 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:09.282294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:09.282345 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:09.282391 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:09.282809 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.283495 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.284828 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.285383 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.285470 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.285512 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:09.285917 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 20:13:09.285988 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260926 20:13:09.286424 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.279333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:41785 (local address 127.23.50.131:53721)
0926 20:13:09.282025 (+ 2692us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:09.282040 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:09.282174 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:09.282517 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:09.282522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:09.282543 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:09.282659 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.282667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.283633 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.283636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.284675 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.284686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.285623 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.285627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.285721 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.285763 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:13:09.286229 (+ 466us) client_negotiation.cc:240] Negotiation successful
0926 20:13:09.286302 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2629,"thread_start_us":105,"threads_started":1}
I20260926 20:13:09.286640 5094 catalog_manager.cc:5495] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1a6c17d045c429e9609fdf788430a59 (127.23.50.130). New cstate: current_term: 1 leader_uuid: "e1a6c17d045c429e9609fdf788430a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20260926 20:13:09.286963 5239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8798329dc44bdaab4542e15b07b96c" candidate_uuid: "0b5d71e02915496594f99d7cc851d253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82531edff95844979c4610ec2b514f66" is_pre_election: true
I20260926 20:13:09.287141 5239 raft_consensus.cc:2379] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b5d71e02915496594f99d7cc851d253 in current term 1: Already voted for candidate e1a6c17d045c429e9609fdf788430a59 in this term.
I20260926 20:13:09.288136 5374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc8798329dc44bdaab4542e15b07b96c" candidate_uuid: "0b5d71e02915496594f99d7cc851d253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a6c17d045c429e9609fdf788430a59" is_pre_election: true
I20260926 20:13:09.288745 5442 leader_election.cc:302] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b5d71e02915496594f99d7cc851d253; no voters: 82531edff95844979c4610ec2b514f66, e1a6c17d045c429e9609fdf788430a59
I20260926 20:13:09.289026 5579 raft_consensus.cc:2726] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260926 20:13:09.290854 5421 tablet.cc:2369] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:13:09.304864 5287 tablet.cc:2369] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:13:09.354375 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.350618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.23.50.130:37309)
0926 20:13:09.350752 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:09.350755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:09.350812 (+ 57us) server_negotiation.cc:407] Connection header received
0926 20:13:09.351046 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:09.351049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:09.351108 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:09.351247 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:13:09.351512 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.352012 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.352953 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.353175 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.353293 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.353564 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:13:09.354034 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:13:09.354129 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:09.354193 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 20:13:09.354265 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:13:09.360491 5239 raft_consensus.cc:1261] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Refusing update from remote peer e1a6c17d045c429e9609fdf788430a59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:13:09.360520 5508 raft_consensus.cc:1261] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [term 1 FOLLOWER]: Refusing update from remote peer e1a6c17d045c429e9609fdf788430a59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:13:09.361155 5582 consensus_queue.cc:1035] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:13:09.361330 5577 consensus_queue.cc:1035] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:13:09.370889 5586 mvcc.cc:205] Tried to move back new op lower bound from 7333697902012760064 to 7333697901708353536. Current Snapshot: MvccSnapshot[applied={T|T < 7333697902012760064}]
I20260926 20:13:09.377661 5094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:13:09.377897 5094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:13:09.380383 5094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Sending DeleteTablet for 3 replicas of tablet cc8798329dc44bdaab4542e15b07b96c
I20260926 20:13:09.381181 5488 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8798329dc44bdaab4542e15b07b96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:09 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:50678
I20260926 20:13:09.381270 5220 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8798329dc44bdaab4542e15b07b96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:09 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:51300
I20260926 20:13:09.381271 5354 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8798329dc44bdaab4542e15b07b96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:09 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:40904
I20260926 20:13:09.381739 5596 tablet_replica.cc:330] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253: stopping tablet replica
I20260926 20:13:09.381771 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 5157
I20260926 20:13:09.381795 5597 tablet_replica.cc:330] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: stopping tablet replica
I20260926 20:13:09.382079 5596 raft_consensus.cc:2229] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:13:09.382117 5597 raft_consensus.cc:2229] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:13:09.382378 5596 raft_consensus.cc:2258] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:09.382486 5597 raft_consensus.cc:2258] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:09.384097 5596 ts_tablet_manager.cc:1905] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:09.388187 5596 ts_tablet_manager.cc:1918] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:13:09.389791 5597 ts_tablet_manager.cc:1905] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:09.389998 5596 log.cc:1196] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/cc8798329dc44bdaab4542e15b07b96c
I20260926 20:13:09.390457 5596 ts_tablet_manager.cc:1939] T cc8798329dc44bdaab4542e15b07b96c P 0b5d71e02915496594f99d7cc851d253: Deleting consensus metadata
W20260926 20:13:09.391665 5080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:41785 (error 108)
I20260926 20:13:09.391933 5079 catalog_manager.cc:4841] TS 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255): tablet cc8798329dc44bdaab4542e15b07b96c (table test-table [id=e201128c2a4b42c4a2e9cf48fe69da20]) successfully deleted
I20260926 20:13:09.392477 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 5291
I20260926 20:13:09.394198 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.393902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:41785 (local address 127.0.0.1:51308)
0926 20:13:09.394013 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:09.394094 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 20:13:09.394400 5080 catalog_manager.cc:4584] TS 82531edff95844979c4610ec2b514f66 (127.23.50.129:41785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc8798329dc44bdaab4542e15b07b96c: Network error: Client connection negotiation failed: client connection to 127.23.50.129:41785: connect: Connection refused (error 111)
I20260926 20:13:09.399019 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 5425
I20260926 20:13:09.399183 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.398909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:37309 (local address 127.0.0.1:40922)
0926 20:13:09.399022 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:09.399083 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.130:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 20:13:09.404422 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 5063
I20260926 20:13:09.410770 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37837
--webserver_interface=127.23.50.190
--webserver_port=38029
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:37837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:09.520953 5599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:09.521252 5599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:09.521297 5599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:09.525388 5599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:13:09.525447 5599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:09.525467 5599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:09.525517 5599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:13:09.525550 5599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:13:09.531111 5599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:37837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=38029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:09.532238 5599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:09.533283 5599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:13:09.551406 5599 server_base.cc:1034] running on GCE node
W20260926 20:13:09.551331 5606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:09.551316 5608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:09.551321 5605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:09.552176 5599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:09.553442 5599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:09.554615 5599 hybrid_clock.cc:648] HybridClock initialized: now 1790453589554597 us; error 32 us; skew 500 ppm
Sep 26 20:13:09 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453589, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:09.557608 5599 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:13:09.558648 5599 webserver.cc:460] Webserver started at http://127.23.50.190:38029/ using document root <none> and password file <none>
I20260926 20:13:09.559250 5599 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:09.559356 5599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:09.563259 5599 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:13:09.565418 5615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:09.566685 5599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:13:09.566849 5599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "a8e2fc54aee34b0491de18484326bdbf"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "12e14180ece8b02ba4ad686d700f274b"
server_key_iv: "aaf4a5b3146098f89f5dfc7fd02c3784"
server_key_version: "encryptionkey@0"
I20260926 20:13:09.567463 5599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:09.598985 5599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:09.601995 5599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:09.602224 5599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:09.609756 5599 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:37837
I20260926 20:13:09.610108 5667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:37837 every 8 connection(s)
I20260926 20:13:09.610982 5599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:13:09.615573 5668 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:13:09.617276 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 5599
I20260926 20:13:09.617403 5668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Bootstrap starting.
I20260926 20:13:09.617775 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:41785
--local_ip_for_outbound_sockets=127.23.50.129
--tserver_master_addrs=127.23.50.190:37837
--webserver_port=44147
--webserver_interface=127.23.50.129
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:13:09.635462 5668 log.cc:824] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:09.644269 5668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:13:09.644654 5668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Bootstrap complete.
I20260926 20:13:09.648824 5668 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } }
I20260926 20:13:09.649216 5668 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8e2fc54aee34b0491de18484326bdbf, State: Initialized, Role: FOLLOWER
I20260926 20:13:09.649760 5668 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } }
I20260926 20:13:09.649953 5668 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:13:09.650032 5668 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:13:09.650153 5668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:13:09.652230 5668 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } }
I20260926 20:13:09.652587 5668 leader_election.cc:302] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8e2fc54aee34b0491de18484326bdbf; no voters:
I20260926 20:13:09.652936 5668 leader_election.cc:288] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:13:09.653024 5672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:13:09.653252 5672 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 2 LEADER]: Becoming Leader. State: Replica: a8e2fc54aee34b0491de18484326bdbf, State: Running, Role: LEADER
I20260926 20:13:09.653627 5672 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } }
I20260926 20:13:09.654229 5668 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:13:09.655128 5674 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8e2fc54aee34b0491de18484326bdbf. Latest consensus state: current_term: 2 leader_uuid: "a8e2fc54aee34b0491de18484326bdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } } }
I20260926 20:13:09.655251 5674 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:09.655180 5673 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a8e2fc54aee34b0491de18484326bdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } } }
I20260926 20:13:09.655319 5673 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:09.655634 5680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:13:09.660634 5680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:13:09.661366 5680 catalog_manager.cc:1259] Loaded cluster ID: e490f6a2b7dd4274b8d468d0ebe75d9e
I20260926 20:13:09.661448 5680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:13:09.662878 5680 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 20:13:09.665620 5680 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 20:13:09.670734 5680 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:13:09.671460 5680 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Loaded TSK: 0
I20260926 20:13:09.672607 5680 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:13:09.746337 5670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:09.746626 5670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:09.746708 5670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:09.751210 5670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:09.751295 5670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:09.751358 5670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:13:09.757419 5670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:41785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=44147
--tserver_master_addrs=127.23.50.190:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:09.758558 5670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:09.759593 5670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:09.769546 5695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:09.769510 5696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:09.769512 5698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:09.769794 5670 server_base.cc:1034] running on GCE node
I20260926 20:13:09.770352 5670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:09.771090 5670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:09.772361 5670 hybrid_clock.cc:648] HybridClock initialized: now 1790453589772313 us; error 61 us; skew 500 ppm
Sep 26 20:13:09 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453589, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:09.775221 5670 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:13:09.776237 5670 webserver.cc:460] Webserver started at http://127.23.50.129:44147/ using document root <none> and password file <none>
I20260926 20:13:09.776794 5670 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:09.776867 5670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:09.780972 5670 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:13:09.783231 5705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:09.784606 5670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:13:09.784742 5670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "82531edff95844979c4610ec2b514f66"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "255214cff6f7a25daa59f2b4d911e312"
server_key_iv: "ca4b0b17050971bd73c27494019c8fc0"
server_key_version: "encryptionkey@0"
I20260926 20:13:09.785343 5670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:09.805550 5670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:09.809145 5670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:09.809361 5670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:09.810117 5670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:09.811293 5712 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:13:09.815645 5670 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:13:09.815723 5670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260926 20:13:09.815776 5670 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:13:09.817430 5670 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:13:09.817493 5670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 20:13:09.817759 5712 tablet_bootstrap.cc:492] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Bootstrap starting.
I20260926 20:13:09.828938 5670 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:41785
I20260926 20:13:09.828997 5819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:41785 every 8 connection(s)
I20260926 20:13:09.830313 5670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:13:09.835306 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 5670
I20260926 20:13:09.835808 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:37309
--local_ip_for_outbound_sockets=127.23.50.130
--tserver_master_addrs=127.23.50.190:37837
--webserver_port=44513
--webserver_interface=127.23.50.130
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:13:09.839736 5712 log.cc:824] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:13:09 dist-test-slave-7034 krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453589, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:09.848194 5712 tablet_bootstrap.cc:492] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:13:09.848706 5712 tablet_bootstrap.cc:492] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Bootstrap complete.
I20260926 20:13:09.850112 5712 ts_tablet_manager.cc:1397] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.011s
I20260926 20:13:09.851163 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.832621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:42441 (local address 127.23.50.190:37837)
0926 20:13:09.832992 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:09.833014 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:09.833361 (+ 347us) server_negotiation.cc:407] Connection header received
0926 20:13:09.834607 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:09.834618 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:09.834928 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:09.835382 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:09.837056 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.838764 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.840706 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.841410 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.841597 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.845036 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:09.845045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:09.845062 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:09.845099 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:09.847309 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:09.847952 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:09.847959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:09.847965 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:09.848146 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:09.848539 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:09.848543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:09.848545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:09.849034 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:09.849182 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:09.849962 (+ 780us) server_negotiation.cc:299] Negotiation successful
0926 20:13:09.850272 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":147,"threads_started":1}
I20260926 20:13:09.851300 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.832634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:42441)
0926 20:13:09.833196 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:09.833239 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:09.834144 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:09.835473 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:09.835485 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:09.836112 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:09.836808 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.836830 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.838954 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.838961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.840550 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.840568 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.841587 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.841594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.841877 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.842643 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:09.842669 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:09.844787 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:09.847448 (+ 2661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:09.847454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:09.847469 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:09.847818 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:09.848254 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:09.848257 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:09.848259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:09.848428 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:09.849242 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:09.849251 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:09.849554 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 20:13:09.849788 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 20:13:09.850172 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":163,"threads_started":1}
I20260926 20:13:09.853052 5712 raft_consensus.cc:348] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:09.853703 5712 raft_consensus.cc:729] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82531edff95844979c4610ec2b514f66, State: Initialized, Role: FOLLOWER
I20260926 20:13:09.854285 5820 heartbeater.cc:347] Connected to a master server at 127.23.50.190:37837
I20260926 20:13:09.854316 5712 consensus_queue.cc:260] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:09.854516 5820 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:09.855124 5820 heartbeater.cc:511] Master 127.23.50.190:37837 requested a full tablet report, sending...
I20260926 20:13:09.855522 5712 ts_tablet_manager.cc:1428] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:13:09.857775 5632 ts_manager.cc:194] Registered new tserver with Master: 82531edff95844979c4610ec2b514f66 (127.23.50.129:41785)
I20260926 20:13:09.861321 5632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:42441
I20260926 20:13:09.864033 5820 heartbeater.cc:503] Master 127.23.50.190:37837 was elected leader, sending a full tablet report...
Sep 26 20:13:09 dist-test-slave-7034 krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453589, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.129@KRBTEST.COM
I20260926 20:13:09.873502 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.860670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:41785 (local address 127.0.0.1:51312)
0926 20:13:09.861446 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:09.861494 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:09.861642 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:09.862308 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:09.862314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:09.864570 (+ 2256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:09.864711 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.864720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.866072 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.866075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.866712 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:09.866721 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.867470 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.867473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:09.867576 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.868224 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:09.868236 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:09.869602 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:09.871811 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:09.871818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:09.871823 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:09.872062 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:09.872374 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:09.872376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:09.872377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:09.872465 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:09.873005 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:09.873010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:09.873111 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 20:13:09.873219 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 20:13:09.873369 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":124,"threads_started":1}
I20260926 20:13:09.873718 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:09.860794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.23.50.129:41785)
0926 20:13:09.861184 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:09.861196 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:09.861603 (+ 407us) server_negotiation.cc:407] Connection header received
0926 20:13:09.861796 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:09.861802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:09.862033 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:09.862209 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:09.864844 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.865963 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.866838 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:09.867327 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:09.867439 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:09.869747 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:09.869758 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:09.869770 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:09.869803 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:09.871679 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:09.872172 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:09.872178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:09.872180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:09.872272 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:09.872547 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:09.872551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:09.872552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:09.872884 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:09.873008 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:09.873347 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 20:13:09.873542 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":111,"threads_started":1}
I20260926 20:13:09.874709 5754 tablet_service.cc:1514] Processing DeleteTablet for tablet cc8798329dc44bdaab4542e15b07b96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:09 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:51312
I20260926 20:13:09.875555 5829 tablet_replica.cc:330] stopping tablet replica
I20260926 20:13:09.875851 5829 raft_consensus.cc:2229] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:13:09.876102 5829 raft_consensus.cc:2258] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:09.877033 5829 ts_tablet_manager.cc:1905] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:09.880561 5829 ts_tablet_manager.cc:1918] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:13:09.880645 5829 log.cc:1196] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal/wals/cc8798329dc44bdaab4542e15b07b96c
I20260926 20:13:09.880939 5829 ts_tablet_manager.cc:1939] T cc8798329dc44bdaab4542e15b07b96c P 82531edff95844979c4610ec2b514f66: Deleting consensus metadata
I20260926 20:13:09.882597 5617 catalog_manager.cc:4841] TS 82531edff95844979c4610ec2b514f66 (127.23.50.129:41785): tablet cc8798329dc44bdaab4542e15b07b96c (table test-table [id=e201128c2a4b42c4a2e9cf48fe69da20]) successfully deleted
W20260926 20:13:09.957175 5824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:09.957485 5824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:09.957561 5824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:09.961767 5824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:09.961853 5824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:09.961968 5824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:13:09.967411 5824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:37309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=44513
--tserver_master_addrs=127.23.50.190:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:09.968534 5824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:09.969583 5824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:09.979728 5835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:09.979768 5834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:09.979766 5824 server_base.cc:1034] running on GCE node
W20260926 20:13:09.979715 5837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:09.980368 5824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:09.981033 5824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:09.982234 5824 hybrid_clock.cc:648] HybridClock initialized: now 1790453589982216 us; error 31 us; skew 500 ppm
Sep 26 20:13:09 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453589, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:09.984987 5824 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:13:09.985986 5824 webserver.cc:460] Webserver started at http://127.23.50.130:44513/ using document root <none> and password file <none>
I20260926 20:13:09.986554 5824 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:09.986658 5824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:09.990628 5824 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 20:13:09.992785 5844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:09.994405 5824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:13:09.994536 5824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "e1a6c17d045c429e9609fdf788430a59"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "d4e3d005143a2995d852e1d419b4ad54"
server_key_iv: "fe5e4bcf357100fca99769afc007b6f5"
server_key_version: "encryptionkey@0"
I20260926 20:13:09.995079 5824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:10.014259 5824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:10.017074 5824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:10.017300 5824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:10.017961 5824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:10.019009 5851 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 20:13:10.022902 5851 ts_tablet_manager.cc:1852] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 20:13:10.023298 5851 ts_tablet_manager.cc:1905] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:10.027196 5851 ts_tablet_manager.cc:1918] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:13:10.027324 5851 log.cc:1196] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/cc8798329dc44bdaab4542e15b07b96c
I20260926 20:13:10.027695 5851 ts_tablet_manager.cc:1939] T cc8798329dc44bdaab4542e15b07b96c P e1a6c17d045c429e9609fdf788430a59: Deleting consensus metadata
I20260926 20:13:10.028018 5824 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 20:13:10.028083 5824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260926 20:13:10.028137 5824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:10.028158 5824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:10.037427 5824 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:37309
I20260926 20:13:10.037460 5958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:37309 every 8 connection(s)
I20260926 20:13:10.038375 5824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:13:10.042815 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 5824
I20260926 20:13:10.043220 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:36255
--local_ip_for_outbound_sockets=127.23.50.131
--tserver_master_addrs=127.23.50.190:37837
--webserver_port=38689
--webserver_interface=127.23.50.131
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:13:10 dist-test-slave-7034 krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453589, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:10.051765 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.040170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:34677 (local address 127.23.50.190:37837)
0926 20:13:10.040344 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:10.040349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:10.041213 (+ 864us) server_negotiation.cc:407] Connection header received
0926 20:13:10.042143 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:10.042146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:10.042197 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:10.042282 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:10.043556 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.044094 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.045215 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.045459 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.045574 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.048202 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:10.048206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:10.048208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:10.048232 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:10.049698 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:10.050253 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:10.050258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:10.050260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:10.050339 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:10.050676 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:10.050679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:10.050681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:10.050867 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:10.050971 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:10.051456 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 20:13:10.051594 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:13:10.052608 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.040601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:34677)
0926 20:13:10.041052 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:10.041098 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:10.041941 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:10.042457 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:10.042464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:10.042799 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:10.043403 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.043429 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.044227 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.044231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.045089 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.045098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.045615 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.045620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.045795 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.046489 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:10.046504 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:10.048014 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:10.049880 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:10.049886 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:10.049901 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:10.050150 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:10.050435 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:10.050439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:10.050441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:10.050576 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:10.050969 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:10.050976 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:10.051217 (+ 241us) client_negotiation.cc:769] Sending connection context
0926 20:13:10.051401 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 20:13:10.051717 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":101,"threads_started":1}
I20260926 20:13:10.053987 5959 heartbeater.cc:347] Connected to a master server at 127.23.50.190:37837
I20260926 20:13:10.054239 5959 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:10.054780 5959 heartbeater.cc:511] Master 127.23.50.190:37837 requested a full tablet report, sending...
I20260926 20:13:10.055953 5632 ts_manager.cc:194] Registered new tserver with Master: e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309)
I20260926 20:13:10.056766 5632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:34677
W20260926 20:13:10.154719 5962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:10.154975 5962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:10.155018 5962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:10.160365 5962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:10.160425 5962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:10.160485 5962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:13:10.165824 5962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:36255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=38689
--tserver_master_addrs=127.23.50.190:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:10.166994 5962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:10.167955 5962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:10.177443 5970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:10.177495 5968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:10.177480 5967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:10.177906 5962 server_base.cc:1034] running on GCE node
I20260926 20:13:10.178335 5962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:10.179005 5962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:10.180166 5962 hybrid_clock.cc:648] HybridClock initialized: now 1790453590180146 us; error 30 us; skew 500 ppm
Sep 26 20:13:10 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:10.182955 5962 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:13:10.183837 5962 webserver.cc:460] Webserver started at http://127.23.50.131:38689/ using document root <none> and password file <none>
I20260926 20:13:10.184391 5962 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:10.184500 5962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:10.188354 5962 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:13:10.190534 5977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:10.191795 5962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:13:10.191926 5962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "0b5d71e02915496594f99d7cc851d253"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "a98a1f65a734d29eb6c7dcd74d207972"
server_key_iv: "8acbc1feaeffe06bf5426b142205254f"
server_key_version: "encryptionkey@0"
I20260926 20:13:10.192477 5962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:10.205590 5962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:10.209079 5962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:10.209304 5962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:10.209983 5962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:10.210867 5962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:13:10.210935 5962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:10.210997 5962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:10.211047 5962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:10.220069 5962 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:36255
I20260926 20:13:10.220088 6090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:36255 every 8 connection(s)
I20260926 20:13:10.221123 5962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:13:10.229828 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 5962
Sep 26 20:13:10 dist-test-slave-7034 krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:10.235846 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.223052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:53293 (local address 127.23.50.190:37837)
0926 20:13:10.223191 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:10.223195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:10.224052 (+ 857us) server_negotiation.cc:407] Connection header received
0926 20:13:10.225479 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:10.225482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:10.225542 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:10.225642 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:10.227045 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.227592 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.228787 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.229014 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.229133 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.232041 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:10.232046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:10.232051 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:10.232078 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:10.233612 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:10.234234 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:10.234238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:10.234240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:10.234329 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:10.234690 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:10.234694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:10.234695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:10.234882 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:10.234991 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:10.235517 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 20:13:10.235642 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:13:10.236923 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.223468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:53293)
0926 20:13:10.223903 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:10.223946 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:10.225232 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:10.225884 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:10.225896 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:10.226322 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:10.226879 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.226895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.227730 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.227738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.228647 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.228656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.229171 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.229175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.229362 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.230064 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:10.230085 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:10.231878 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:10.233736 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:10.233743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:10.233760 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:10.234125 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:10.234425 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:10.234429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:10.234431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:10.234586 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:10.234989 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:10.234996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:10.235261 (+ 265us) client_negotiation.cc:769] Sending connection context
0926 20:13:10.235457 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 20:13:10.235804 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260926 20:13:10.238197 6091 heartbeater.cc:347] Connected to a master server at 127.23.50.190:37837
I20260926 20:13:10.238466 6091 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:10.239100 6091 heartbeater.cc:511] Master 127.23.50.190:37837 requested a full tablet report, sending...
I20260926 20:13:10.239198 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.230583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.23.50.190:37837)
0926 20:13:10.230893 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:10.230899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:10.230965 (+ 66us) server_negotiation.cc:407] Connection header received
0926 20:13:10.231153 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:10.231158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:10.231238 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:10.231331 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:10.232076 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.232882 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.233969 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.234202 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.234313 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.235681 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:10.235685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:10.235688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:10.235718 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:10.237350 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:10.237883 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:10.237887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:10.237888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:10.237976 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:10.238330 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:10.238335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:10.238337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:10.238567 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:10.238725 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:10.238951 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 20:13:10.239052 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
I20260926 20:13:10.240310 5632 ts_manager.cc:194] Registered new tserver with Master: 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255)
I20260926 20:13:10.240921 5632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:53293
I20260926 20:13:10.241086 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:13:10.245241 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.241718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.23.50.190:37837)
0926 20:13:10.241826 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:10.241830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:10.242076 (+ 246us) server_negotiation.cc:407] Connection header received
0926 20:13:10.242182 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:10.242185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:10.242310 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:10.242415 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:13:10.242615 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.243312 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.244085 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.244314 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.244389 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.244631 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:13:10.244965 (+ 334us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:13:10.245049 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:10.245105 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 20:13:10.245152 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:13:10.245786 5632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:13:10.249095 5632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:13:10.256912 5754 tablet_service.cc:1467] Processing CreateTablet for tablet 3b56501612cd4b97829ebb2f16fef3bd (DEFAULT_TABLE table=test-table [id=f3eabd159197413a8d95c0f7657f093b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:10.257581 5754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b56501612cd4b97829ebb2f16fef3bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:10.260846 5712 tablet_bootstrap.cc:492] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66: Bootstrap starting.
I20260926 20:13:10.261835 5712 tablet_bootstrap.cc:654] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:10.262331 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.256076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.23.50.130:37309)
0926 20:13:10.256428 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:10.256440 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:10.256467 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:13:10.256624 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:10.256638 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:10.256821 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:10.257055 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:10.257304 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.258443 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.260551 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.261697 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.261807 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.261998 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:10.262127 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 20:13:10.262201 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":147,"threads_started":1}
I20260926 20:13:10.262378 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.255824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:37309 (local address 127.0.0.1:40934)
0926 20:13:10.255927 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:10.255943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:10.256018 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:10.257036 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:10.257040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:10.257060 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:10.257171 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.257178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.259103 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.259108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.260402 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.260415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.261805 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.261810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.261946 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.262005 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 20:13:10.262088 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:13:10.262193 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":30}
I20260926 20:13:10.263417 5712 tablet_bootstrap.cc:492] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66: No bootstrap required, opened a new log
I20260926 20:13:10.263511 5712 ts_tablet_manager.cc:1397] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:13:10.263967 5712 raft_consensus.cc:348] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:10.264091 5712 raft_consensus.cc:374] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:10.264130 5712 raft_consensus.cc:729] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82531edff95844979c4610ec2b514f66, State: Initialized, Role: FOLLOWER
I20260926 20:13:10.264248 5712 consensus_queue.cc:260] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:10.263965 5893 tablet_service.cc:1467] Processing CreateTablet for tablet 3b56501612cd4b97829ebb2f16fef3bd (DEFAULT_TABLE table=test-table [id=f3eabd159197413a8d95c0f7657f093b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:10.264581 5712 ts_tablet_manager.cc:1428] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:13:10.265111 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b56501612cd4b97829ebb2f16fef3bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:10.265955 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.256260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:36255 (local address 127.0.0.1:50686)
0926 20:13:10.258010 (+ 1750us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:10.258026 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:10.258126 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:10.258638 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:10.258642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:10.258661 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:10.258752 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.258758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.259793 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.259799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.260578 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.260588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.265469 (+ 4881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.265473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.265577 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.265601 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:13:10.265693 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 20:13:10.265769 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1687,"mutex_wait_us":20,"thread_start_us":100,"threads_started":1}
I20260926 20:13:10.266043 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.256674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.23.50.131:36255)
0926 20:13:10.257044 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:10.257056 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:10.258122 (+ 1066us) server_negotiation.cc:407] Connection header received
0926 20:13:10.258257 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:10.258263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:10.258483 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:10.258649 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:10.258849 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.259656 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.262800 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.265335 (+ 2535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.265450 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.265583 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:10.265759 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 20:13:10.265902 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":151,"threads_started":1}
I20260926 20:13:10.269563 6025 tablet_service.cc:1467] Processing CreateTablet for tablet 3b56501612cd4b97829ebb2f16fef3bd (DEFAULT_TABLE table=test-table [id=f3eabd159197413a8d95c0f7657f093b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:10.270653 6025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b56501612cd4b97829ebb2f16fef3bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:10.274153 5851 tablet_bootstrap.cc:492] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59: Bootstrap starting.
I20260926 20:13:10.276005 5851 tablet_bootstrap.cc:654] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:10.276772 5851 log.cc:824] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:10.277160 6104 tablet_bootstrap.cc:492] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253: Bootstrap starting.
I20260926 20:13:10.278645 5851 tablet_bootstrap.cc:492] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59: No bootstrap required, opened a new log
I20260926 20:13:10.278821 5851 ts_tablet_manager.cc:1397] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:13:10.279119 6104 tablet_bootstrap.cc:654] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:10.280112 6104 log.cc:824] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:10.282284 6104 tablet_bootstrap.cc:492] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253: No bootstrap required, opened a new log
I20260926 20:13:10.282470 6104 ts_tablet_manager.cc:1397] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:13:10.282406 5851 raft_consensus.cc:348] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:10.282668 5851 raft_consensus.cc:374] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:10.282729 5851 raft_consensus.cc:729] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a6c17d045c429e9609fdf788430a59, State: Initialized, Role: FOLLOWER
I20260926 20:13:10.283188 5851 consensus_queue.cc:260] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:10.283895 5959 heartbeater.cc:503] Master 127.23.50.190:37837 was elected leader, sending a full tablet report...
I20260926 20:13:10.284171 5851 ts_tablet_manager.cc:1428] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 20:13:10.285794 6104 raft_consensus.cc:348] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:10.286087 6104 raft_consensus.cc:374] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:10.286188 6104 raft_consensus.cc:729] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5d71e02915496594f99d7cc851d253, State: Initialized, Role: FOLLOWER
I20260926 20:13:10.286625 6104 consensus_queue.cc:260] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:10.287310 6091 heartbeater.cc:503] Master 127.23.50.190:37837 was elected leader, sending a full tablet report...
I20260926 20:13:10.287521 6104 ts_tablet_manager.cc:1428] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 20:13:10.289307 5960 tablet.cc:2369] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:13:10.331760 5821 tablet.cc:2369] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:13:10.462464 6108 raft_consensus.cc:482] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:10.462699 6108 raft_consensus.cc:504] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:10.463913 6108 leader_election.cc:288] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309), 82531edff95844979c4610ec2b514f66 (127.23.50.129:41785)
I20260926 20:13:10.467265 6107 raft_consensus.cc:482] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:10.467450 6107 raft_consensus.cc:504] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:10.468250 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.464200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:37309 (local address 127.23.50.131:39495)
0926 20:13:10.464363 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:10.464397 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:10.464514 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:10.465081 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:10.465084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:10.465101 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:10.465211 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.465224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.466023 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.466027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.466878 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.466887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.467821 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.467824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.467930 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.467952 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:13:10.468024 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:13:10.468117 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":22}
I20260926 20:13:10.468489 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.464556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:41785 (local address 127.23.50.131:43047)
0926 20:13:10.464931 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:10.464943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:10.465030 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:10.465364 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:10.465367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:10.465383 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:10.465488 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.465493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.466311 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.466317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.467271 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.467281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.468083 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.468087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.468187 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.468207 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 20:13:10.468282 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:13:10.468347 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":19,"thread_start_us":165,"threads_started":1}
I20260926 20:13:10.468482 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.464689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:43047 (local address 127.23.50.129:41785)
0926 20:13:10.464945 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:10.464949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:10.465002 (+ 53us) server_negotiation.cc:407] Connection header received
0926 20:13:10.465151 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:10.465154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:10.465221 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:10.465330 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:10.465596 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.466206 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.467387 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.467949 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.468096 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.468253 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:10.468295 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 20:13:10.468361 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
I20260926 20:13:10.468799 6107 leader_election.cc:288] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255), 82531edff95844979c4610ec2b514f66 (127.23.50.129:41785)
I20260926 20:13:10.468986 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.464342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:39495 (local address 127.23.50.130:37309)
0926 20:13:10.464487 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:10.464492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:10.464508 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:13:10.464917 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:10.464922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:10.464982 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:10.465068 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:10.465335 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.465913 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.467006 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.467583 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.467697 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.467748 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:10.468792 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0926 20:13:10.468870 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:13:10.469333 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b56501612cd4b97829ebb2f16fef3bd" candidate_uuid: "0b5d71e02915496594f99d7cc851d253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82531edff95844979c4610ec2b514f66" is_pre_election: true
I20260926 20:13:10.469497 5913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b56501612cd4b97829ebb2f16fef3bd" candidate_uuid: "0b5d71e02915496594f99d7cc851d253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a6c17d045c429e9609fdf788430a59" is_pre_election: true
I20260926 20:13:10.469563 5774 raft_consensus.cc:2454] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b5d71e02915496594f99d7cc851d253 in term 0.
I20260926 20:13:10.469643 5913 raft_consensus.cc:2454] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b5d71e02915496594f99d7cc851d253 in term 0.
I20260926 20:13:10.470031 5979 leader_election.cc:302] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5d71e02915496594f99d7cc851d253, e1a6c17d045c429e9609fdf788430a59; no voters:
I20260926 20:13:10.470340 6108 raft_consensus.cc:2781] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:13:10.470468 6108 raft_consensus.cc:482] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:13:10.470561 6108 raft_consensus.cc:3037] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 0 FOLLOWER]: Advancing to term 1
W20260926 20:13:10.472043 6092 tablet.cc:2369] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:13:10.471992 6108 raft_consensus.cc:504] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:10.472734 5913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b56501612cd4b97829ebb2f16fef3bd" candidate_uuid: "0b5d71e02915496594f99d7cc851d253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a6c17d045c429e9609fdf788430a59"
I20260926 20:13:10.472859 5913 raft_consensus.cc:3037] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:10.473081 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b56501612cd4b97829ebb2f16fef3bd" candidate_uuid: "0b5d71e02915496594f99d7cc851d253" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82531edff95844979c4610ec2b514f66"
I20260926 20:13:10.473214 5774 raft_consensus.cc:3037] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:10.473830 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.468956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:36255 (local address 127.23.50.130:57677)
0926 20:13:10.469909 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:10.469924 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:10.470005 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:10.470328 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:10.470331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:10.470351 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:10.470460 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.470466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.471324 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.471330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.472440 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.472450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.473387 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.473391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.473502 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.473524 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:13:10.473598 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:13:10.473667 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"spinlock_wait_cycles":2432}
I20260926 20:13:10.474156 6111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.469093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:41785 (local address 127.23.50.130:34931)
0926 20:13:10.470255 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:10.470268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:10.470343 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:10.470660 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:10.470663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:10.470677 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:10.470804 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.470810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.471711 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.471714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.472814 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:10.472822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.473730 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.473733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:10.473824 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.473880 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 20:13:10.473954 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 20:13:10.474030 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"thread_start_us":119,"threads_started":1}
I20260926 20:13:10.474193 5913 raft_consensus.cc:2454] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b5d71e02915496594f99d7cc851d253 in term 1.
I20260926 20:13:10.474259 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.469143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:57677 (local address 127.23.50.131:36255)
0926 20:13:10.469254 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:10.469260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:10.469972 (+ 712us) server_negotiation.cc:407] Connection header received
0926 20:13:10.470109 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:10.470112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:10.470157 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:10.470222 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:10.470571 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.471205 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.472624 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.473191 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.473291 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.473333 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:10.474080 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 20:13:10.474146 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:13:10.474565 5774 raft_consensus.cc:2454] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b5d71e02915496594f99d7cc851d253 in term 1.
I20260926 20:13:10.474604 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.469904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:34931 (local address 127.23.50.129:41785)
0926 20:13:10.470223 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:10.470228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:10.470415 (+ 187us) server_negotiation.cc:407] Connection header received
0926 20:13:10.470459 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:10.470462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:10.470509 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:10.470575 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:10.470925 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.471589 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.472939 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.473539 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.473638 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.473677 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:10.474411 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 20:13:10.474486 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260926 20:13:10.474718 6045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b56501612cd4b97829ebb2f16fef3bd" candidate_uuid: "e1a6c17d045c429e9609fdf788430a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5d71e02915496594f99d7cc851d253" is_pre_election: true
I20260926 20:13:10.474913 6045 raft_consensus.cc:2379] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e1a6c17d045c429e9609fdf788430a59 in current term 1: Already voted for candidate 0b5d71e02915496594f99d7cc851d253 in this term.
I20260926 20:13:10.474963 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b56501612cd4b97829ebb2f16fef3bd" candidate_uuid: "e1a6c17d045c429e9609fdf788430a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82531edff95844979c4610ec2b514f66" is_pre_election: true
I20260926 20:13:10.475039 5979 leader_election.cc:302] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5d71e02915496594f99d7cc851d253, 82531edff95844979c4610ec2b514f66; no voters:
I20260926 20:13:10.475133 5774 raft_consensus.cc:2379] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e1a6c17d045c429e9609fdf788430a59 in current term 1: Already voted for candidate 0b5d71e02915496594f99d7cc851d253 in this term.
I20260926 20:13:10.475203 6108 leader_election.cc:288] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [CANDIDATE]: Term 1 election: Requested vote from peers e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309), 82531edff95844979c4610ec2b514f66 (127.23.50.129:41785)
I20260926 20:13:10.475297 6108 raft_consensus.cc:2781] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:13:10.475415 6108 raft_consensus.cc:686] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 1 LEADER]: Becoming Leader. State: Replica: 0b5d71e02915496594f99d7cc851d253, State: Running, Role: LEADER
I20260926 20:13:10.475600 5846 leader_election.cc:302] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1a6c17d045c429e9609fdf788430a59; no voters: 0b5d71e02915496594f99d7cc851d253, 82531edff95844979c4610ec2b514f66
I20260926 20:13:10.475792 6108 consensus_queue.cc:237] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:10.475876 6107 raft_consensus.cc:2726] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 20:13:10.478899 5632 catalog_manager.cc:5495] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b5d71e02915496594f99d7cc851d253 (127.23.50.131). New cstate: current_term: 1 leader_uuid: "0b5d71e02915496594f99d7cc851d253" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } health_report { overall_health: UNKNOWN } } }
I20260926 20:13:10.505877 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.502521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56052 (local address 127.23.50.131:36255)
0926 20:13:10.502637 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:10.502641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:10.502654 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:13:10.502803 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:10.502805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:10.502859 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:10.502944 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:13:10.503148 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.503648 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.504465 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:10.504673 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:10.504752 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:10.505060 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:13:10.505469 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:13:10.505555 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:10.505615 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 20:13:10.505663 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:13:10.511687 5774 raft_consensus.cc:1261] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Refusing update from remote peer 0b5d71e02915496594f99d7cc851d253: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:13:10.511713 5913 raft_consensus.cc:1261] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [term 1 FOLLOWER]: Refusing update from remote peer 0b5d71e02915496594f99d7cc851d253: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:13:10.512312 6108 consensus_queue.cc:1035] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:13:10.512487 6112 consensus_queue.cc:1035] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:13:10.518664 6116 mvcc.cc:205] Tried to move back new op lower bound from 7333697906728071168 to 7333697906591531008. Current Snapshot: MvccSnapshot[applied={T|T < 7333697906728071168}]
I20260926 20:13:10.520954 6117 mvcc.cc:205] Tried to move back new op lower bound from 7333697906728071168 to 7333697906591531008. Current Snapshot: MvccSnapshot[applied={T|T < 7333697906728071168}]
I20260926 20:13:10.527611 5632 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:13:10.527779 5632 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:13:10.530324 5632 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Sending DeleteTablet for 3 replicas of tablet 3b56501612cd4b97829ebb2f16fef3bd
I20260926 20:13:10.531186 5893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b56501612cd4b97829ebb2f16fef3bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:10 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:40934
I20260926 20:13:10.531183 6025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b56501612cd4b97829ebb2f16fef3bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:10 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:50686
I20260926 20:13:10.531261 5754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b56501612cd4b97829ebb2f16fef3bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:10 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:51312
I20260926 20:13:10.531823 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 5670
I20260926 20:13:10.531841 6126 tablet_replica.cc:330] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253: stopping tablet replica
I20260926 20:13:10.532186 6126 raft_consensus.cc:2229] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:13:10.532565 6126 raft_consensus.cc:2258] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:10.533243 6127 tablet_replica.cc:330] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59: stopping tablet replica
I20260926 20:13:10.533609 6127 raft_consensus.cc:2229] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:13:10.534024 6127 raft_consensus.cc:2258] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:10.534504 6126 ts_tablet_manager.cc:1905] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:10.535419 6127 ts_tablet_manager.cc:1905] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:10.538234 6127 ts_tablet_manager.cc:1918] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:13:10.538365 6127 log.cc:1196] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/3b56501612cd4b97829ebb2f16fef3bd
I20260926 20:13:10.538695 6127 ts_tablet_manager.cc:1939] T 3b56501612cd4b97829ebb2f16fef3bd P e1a6c17d045c429e9609fdf788430a59: Deleting consensus metadata
I20260926 20:13:10.538704 6126 ts_tablet_manager.cc:1918] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:13:10.538944 6126 log.cc:1196] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/3b56501612cd4b97829ebb2f16fef3bd
I20260926 20:13:10.539369 6126 ts_tablet_manager.cc:1939] T 3b56501612cd4b97829ebb2f16fef3bd P 0b5d71e02915496594f99d7cc851d253: Deleting consensus metadata
I20260926 20:13:10.540120 5617 catalog_manager.cc:4841] TS e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309): tablet 3b56501612cd4b97829ebb2f16fef3bd (table test-table [id=f3eabd159197413a8d95c0f7657f093b]) successfully deleted
I20260926 20:13:10.540746 5616 catalog_manager.cc:4841] TS 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255): tablet 3b56501612cd4b97829ebb2f16fef3bd (table test-table [id=f3eabd159197413a8d95c0f7657f093b]) successfully deleted
W20260926 20:13:10.541671 5617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:41785 (error 108)
I20260926 20:13:10.542690 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 5824
I20260926 20:13:10.542820 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:10.542504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:41785 (local address 127.0.0.1:48568)
0926 20:13:10.542643 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:10.542720 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:41785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 20:13:10.543679 5617 catalog_manager.cc:4584] TS 82531edff95844979c4610ec2b514f66 (127.23.50.129:41785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b56501612cd4b97829ebb2f16fef3bd: Network error: Client connection negotiation failed: client connection to 127.23.50.129:41785: connect: Connection refused (error 111)
I20260926 20:13:10.551651 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 5962
I20260926 20:13:10.557312 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 5599
I20260926 20:13:10.564383 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37837
--webserver_interface=127.23.50.190
--webserver_port=38029
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:37837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:10.677933 6129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:10.678254 6129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:10.678323 6129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:10.682324 6129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:13:10.682417 6129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:10.682456 6129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:10.682493 6129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:13:10.682528 6129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:13:10.687809 6129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:37837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=38029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:10.689013 6129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:10.690102 6129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:10.698422 6134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:10.698484 6129 server_base.cc:1034] running on GCE node
W20260926 20:13:10.698505 6137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:10.698505 6135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:10.699177 6129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:10.700274 6129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:10.701465 6129 hybrid_clock.cc:648] HybridClock initialized: now 1790453590701452 us; error 26 us; skew 500 ppm
Sep 26 20:13:10 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:10.704396 6129 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:13:10.705322 6129 webserver.cc:460] Webserver started at http://127.23.50.190:38029/ using document root <none> and password file <none>
I20260926 20:13:10.705948 6129 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:10.706039 6129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:10.709770 6129 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:13:10.712003 6144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:10.713194 6129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:13:10.713333 6129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "a8e2fc54aee34b0491de18484326bdbf"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "12e14180ece8b02ba4ad686d700f274b"
server_key_iv: "aaf4a5b3146098f89f5dfc7fd02c3784"
server_key_version: "encryptionkey@0"
I20260926 20:13:10.713907 6129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:10.732012 6129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:10.735680 6129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:10.735917 6129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:10.743579 6129 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:37837
I20260926 20:13:10.743880 6196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:37837 every 8 connection(s)
I20260926 20:13:10.744844 6129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:13:10.749261 6197 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:13:10.751050 6197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Bootstrap starting.
I20260926 20:13:10.751086 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 6129
I20260926 20:13:10.751556 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:41785
--local_ip_for_outbound_sockets=127.23.50.129
--tserver_master_addrs=127.23.50.190:37837
--webserver_port=44147
--webserver_interface=127.23.50.129
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:13:10.768719 6197 log.cc:824] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:10.779837 6197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:13:10.780232 6197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf: Bootstrap complete.
I20260926 20:13:10.784235 6197 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } }
I20260926 20:13:10.784655 6197 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8e2fc54aee34b0491de18484326bdbf, State: Initialized, Role: FOLLOWER
I20260926 20:13:10.785179 6197 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } }
I20260926 20:13:10.785323 6197 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:13:10.785409 6197 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:13:10.785526 6197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:13:10.787424 6197 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } }
I20260926 20:13:10.787758 6197 leader_election.cc:302] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8e2fc54aee34b0491de18484326bdbf; no voters:
I20260926 20:13:10.788136 6197 leader_election.cc:288] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:13:10.788192 6201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:13:10.788376 6201 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [term 3 LEADER]: Becoming Leader. State: Replica: a8e2fc54aee34b0491de18484326bdbf, State: Running, Role: LEADER
I20260926 20:13:10.788734 6201 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } }
I20260926 20:13:10.789335 6197 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:13:10.790163 6202 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8e2fc54aee34b0491de18484326bdbf. Latest consensus state: current_term: 3 leader_uuid: "a8e2fc54aee34b0491de18484326bdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } } }
I20260926 20:13:10.790307 6202 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:10.790585 6203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a8e2fc54aee34b0491de18484326bdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8e2fc54aee34b0491de18484326bdbf" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37837 } } }
I20260926 20:13:10.790723 6203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8e2fc54aee34b0491de18484326bdbf [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:10.791209 6209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:13:10.796401 6209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:13:10.797181 6209 catalog_manager.cc:1259] Loaded cluster ID: e490f6a2b7dd4274b8d468d0ebe75d9e
I20260926 20:13:10.797259 6209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 20:13:10.798173 6209 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb73b1969ad google::LogMessage::Fail() at ??:0
@ 0x7fb73b19988a google::LogMessage::SendToLog() at ??:0
@ 0x7fb73b1964de google::LogMessage::Flush() at ??:0
@ 0x7fb73b197a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb745ad9fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fb745adad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb745ad7579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fb745b1b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb73d3e83d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb73d3e8c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fb73d3ea3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fb73d3d9512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb73ea1d6db start_thread at ??:0
@ 0x7fb739c9461f clone at ??:0
W20260926 20:13:10.910856 6199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:10.911377 6199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:10.911460 6199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:10.918675 6199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:10.918769 6199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:10.918874 6199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:13:10.928438 6199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:41785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=44147
--tserver_master_addrs=127.23.50.190:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:10.930783 6199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:10.932947 6199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:10.945582 6241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:10.945716 6239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:10.945575 6238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:10.945967 6199 server_base.cc:1034] running on GCE node
I20260926 20:13:10.946604 6199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:10.947255 6199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:10.948477 6199 hybrid_clock.cc:648] HybridClock initialized: now 1790453590948442 us; error 52 us; skew 500 ppm
Sep 26 20:13:10 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:10.952276 6199 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:13:10.953507 6199 webserver.cc:460] Webserver started at http://127.23.50.129:44147/ using document root <none> and password file <none>
I20260926 20:13:10.954306 6199 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:10.954406 6199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:10.959983 6199 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260926 20:13:10.962941 6248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:10.964608 6199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 20:13:10.964742 6199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "82531edff95844979c4610ec2b514f66"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "255214cff6f7a25daa59f2b4d911e312"
server_key_iv: "ca4b0b17050971bd73c27494019c8fc0"
server_key_version: "encryptionkey@0"
I20260926 20:13:10.965541 6199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:10.993963 6199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:10.997515 6199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:10.997745 6199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:10.998413 6199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:10.999588 6255 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:13:11.005474 6199 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:13:11.005590 6199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 20:13:11.005707 6199 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:13:11.007542 6199 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:13:11.007618 6199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 20:13:11.007946 6255 tablet_bootstrap.cc:492] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66: Bootstrap starting.
I20260926 20:13:11.018347 6199 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:41785
I20260926 20:13:11.018515 6362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:41785 every 8 connection(s)
I20260926 20:13:11.019408 6199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:13:11.019707 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 6199
I20260926 20:13:11.020205 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:37309
--local_ip_for_outbound_sockets=127.23.50.130
--tserver_master_addrs=127.23.50.190:37837
--webserver_port=44513
--webserver_interface=127.23.50.130
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:13:11.031196 6255 log.cc:824] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:11.038482 6255 tablet_bootstrap.cc:492] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:13:11.038877 6255 tablet_bootstrap.cc:492] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66: Bootstrap complete.
I20260926 20:13:11.040017 6255 ts_tablet_manager.cc:1397] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.008s
I20260926 20:13:11.042615 6255 raft_consensus.cc:348] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:11.043072 6255 raft_consensus.cc:729] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82531edff95844979c4610ec2b514f66, State: Initialized, Role: FOLLOWER
I20260926 20:13:11.043591 6255 consensus_queue.cc:260] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:11.044590 6255 ts_tablet_manager.cc:1428] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260926 20:13:11.079013 6366 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:13:11.022467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:43403)
0926 20:13:11.022987 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:11.023031 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:11.023785 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:11.078251 (+ 54466us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":149,"threads_started":1}
W20260926 20:13:11.079645 6363 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:37837 (0 consecutive failures): Network error: Failed to ping master at 127.23.50.190:37837: Client connection negotiation failed: client connection to 127.23.50.190:37837: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:13:11.080542 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:11.080230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:52919)
0926 20:13:11.080382 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:11.080473 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 20:13:11.081363 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:11.081133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:35725)
0926 20:13:11.081254 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:11.081313 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 20:13:11.081583 6363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:13:11.160478 6365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:11.160830 6365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:11.160912 6365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:11.165712 6365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:11.165781 6365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:11.165882 6365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:13:11.171329 6365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:37309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=44513
--tserver_master_addrs=127.23.50.190:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:11.172451 6365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:11.173555 6365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:11.183578 6374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:11.183575 6373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:11.183774 6365 server_base.cc:1034] running on GCE node
W20260926 20:13:11.183576 6376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:11.184350 6365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:11.185204 6365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:11.186550 6365 hybrid_clock.cc:648] HybridClock initialized: now 1790453591186532 us; error 70 us; skew 500 ppm
Sep 26 20:13:11 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453591, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:11.189427 6365 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:13:11.190478 6365 webserver.cc:460] Webserver started at http://127.23.50.130:44513/ using document root <none> and password file <none>
I20260926 20:13:11.191053 6365 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:11.191146 6365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:11.194989 6365 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 20:13:11.196995 6383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:11.198338 6365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:13:11.198467 6365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "e1a6c17d045c429e9609fdf788430a59"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "d4e3d005143a2995d852e1d419b4ad54"
server_key_iv: "fe5e4bcf357100fca99769afc007b6f5"
server_key_version: "encryptionkey@0"
I20260926 20:13:11.199013 6365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:11.216717 6365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:11.219877 6365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:11.220072 6365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:11.220686 6365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:11.221508 6365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:13:11.221575 6365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:11.221645 6365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:11.221683 6365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:11.230674 6365 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:37309
I20260926 20:13:11.230690 6496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:37309 every 8 connection(s)
I20260926 20:13:11.231781 6365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:13:11.235801 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:11.233974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:33971)
0926 20:13:11.234497 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:11.234630 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
W20260926 20:13:11.236272 6497 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:37837 (0 consecutive failures): Network error: Failed to ping master at 127.23.50.190:37837: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
I20260926 20:13:11.236845 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:11.236634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:33747)
0926 20:13:11.236732 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:11.236793 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 20:13:11.237358 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:11.237192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:47207)
0926 20:13:11.237276 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:11.237316 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 20:13:11.237520 6497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:13:11.241166 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 6365
I20260926 20:13:11.241603 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:36255
--local_ip_for_outbound_sockets=127.23.50.131
--tserver_master_addrs=127.23.50.190:37837
--webserver_port=38689
--webserver_interface=127.23.50.131
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:11.351207 6500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:11.351493 6500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:11.351565 6500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:11.355595 6500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:11.355667 6500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:11.355780 6500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:13:11.360993 6500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:36255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=38689
--tserver_master_addrs=127.23.50.190:37837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:11.362043 6500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:11.363026 6500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:11.372664 6508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:11.372656 6506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:11.372650 6505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:11.372949 6500 server_base.cc:1034] running on GCE node
I20260926 20:13:11.373450 6500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:11.374362 6500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:11.375574 6500 hybrid_clock.cc:648] HybridClock initialized: now 1790453591375538 us; error 49 us; skew 500 ppm
Sep 26 20:13:11 dist-test-slave-7034 krb5kdc[5050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453591, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:11.378365 6500 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:13:11.379393 6500 webserver.cc:460] Webserver started at http://127.23.50.131:38689/ using document root <none> and password file <none>
I20260926 20:13:11.379984 6500 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:11.380075 6500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:11.383915 6500 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 20:13:11.385982 6515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:11.387183 6500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:13:11.387373 6500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "0b5d71e02915496594f99d7cc851d253"
format_stamp: "Formatted at 2026-09-26 20:13:08 on dist-test-slave-7034"
server_key: "a98a1f65a734d29eb6c7dcd74d207972"
server_key_iv: "8acbc1feaeffe06bf5426b142205254f"
server_key_version: "encryptionkey@0"
I20260926 20:13:11.387912 6500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:11.403847 6500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:11.406615 6500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:11.406817 6500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:11.407420 6500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:11.408252 6500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:13:11.408320 6500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:11.408391 6500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:11.408430 6500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:11.417284 6500 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:36255
I20260926 20:13:11.417301 6628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:36255 every 8 connection(s)
I20260926 20:13:11.418300 6500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:13:11.422142 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:11.420351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:47611)
0926 20:13:11.420801 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:11.420918 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
W20260926 20:13:11.422564 6629 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:37837 (0 consecutive failures): Network error: Failed to ping master at 127.23.50.190:37837: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
I20260926 20:13:11.423153 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:11.422937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:41351)
0926 20:13:11.423045 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:11.423094 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 20:13:11.423704 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:11.423506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:47339)
0926 20:13:11.423607 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:11.423650 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 20:13:11.423873 6629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:13:11.428100 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 6500
I20260926 20:13:12.082691 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:12.082259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:55827)
0926 20:13:12.082517 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:12.082577 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
I20260926 20:13:12.238471 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:12.238077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:60545)
0926 20:13:12.238313 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:12.238370 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260926 20:13:12.424738 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:12.424363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:59465)
0926 20:13:12.424594 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:12.424647 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260926 20:13:12.908720 6636 raft_consensus.cc:482] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:12.908968 6636 raft_consensus.cc:504] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:12.910214 6636 leader_election.cc:288] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309), 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255)
Sep 26 20:13:12 dist-test-slave-7034 krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.131@KRBTEST.COM
Sep 26 20:13:12 dist-test-slave-7034 krb5kdc[5050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453590, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.130@KRBTEST.COM
I20260926 20:13:12.925196 6640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:12.910779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:55311 (local address 127.23.50.131:36255)
0926 20:13:12.911128 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:12.911148 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:12.911179 (+ 31us) server_negotiation.cc:407] Connection header received
0926 20:13:12.911805 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:12.911810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:12.912130 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:12.912370 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:12.913558 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:12.914555 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:12.916389 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:12.917129 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:12.917303 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:12.920686 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:12.920691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:12.920705 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:12.920736 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:12.922656 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:12.923195 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:12.923206 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:12.923211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:12.923323 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:12.923656 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:12.923660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:12.923662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:12.924078 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:12.924227 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:12.924764 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 20:13:12.925043 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":119,"threads_started":1}
I20260926 20:13:12.925189 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:12.910601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:36255 (local address 127.23.50.129:55311)
0926 20:13:12.910927 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:12.910943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:12.911042 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:12.912751 (+ 1709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:12.912753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:12.912974 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:12.913379 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:12.913395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:12.914705 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:12.914708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:12.916255 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:12.916267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:12.917276 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:12.917281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:12.917547 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:12.918325 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:12.918354 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:12.920375 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:12.922794 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:12.922800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:12.922815 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:12.923076 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:12.923418 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:12.923421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:12.923423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:12.923563 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:12.924202 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:12.924209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:12.924457 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 20:13:12.924650 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 20:13:12.924992 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":241,"spinlock_wait_cycles":21376,"thread_start_us":68,"threads_started":1}
I20260926 20:13:12.925299 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:12.910605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:55385 (local address 127.23.50.130:37309)
0926 20:13:12.911028 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:12.911045 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:12.911104 (+ 59us) server_negotiation.cc:407] Connection header received
0926 20:13:12.911792 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:12.911797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:12.912130 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:12.912424 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:12.913544 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:12.914579 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:12.916389 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:12.917129 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:12.917304 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:12.920970 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:12.920975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:12.920987 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:12.921019 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:12.922906 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:12.923399 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:12.923405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:12.923415 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:12.923532 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:12.923845 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:12.923848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:12.923850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:12.924241 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:12.924387 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:12.924781 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 20:13:12.925087 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":71,"threads_started":1}
I20260926 20:13:12.925405 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:12.910407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:37309 (local address 127.23.50.129:55385)
0926 20:13:12.910761 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:12.910776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:12.910885 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:12.912480 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:12.912486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:12.912949 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:12.913379 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:12.913394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:12.914747 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:12.914750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:12.916255 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:12.916267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:12.917276 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:12.917281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:12.917559 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:12.918325 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:12.918358 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:12.920746 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:12.923037 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:12.923042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:12.923045 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:12.923280 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:12.923635 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:12.923640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:12.923642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:12.923734 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:12.924364 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:12.924368 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:12.924500 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 20:13:12.924649 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 20:13:12.925184 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":409,"spinlock_wait_cycles":27264,"thread_start_us":77,"threads_started":1}
W20260926 20:13:12.928489 6250 leader_election.cc:341] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309): Not found: Tablet not found: 3b56501612cd4b97829ebb2f16fef3bd
W20260926 20:13:12.928761 6249 leader_election.cc:341] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255): Not found: Tablet not found: 3b56501612cd4b97829ebb2f16fef3bd
I20260926 20:13:12.928867 6249 leader_election.cc:302] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82531edff95844979c4610ec2b514f66; no voters: 0b5d71e02915496594f99d7cc851d253, e1a6c17d045c429e9609fdf788430a59
I20260926 20:13:12.929203 6636 raft_consensus.cc:2726] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:13:13.083772 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:13.083448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:60159)
0926 20:13:13.083582 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:13.083682 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 20:13:13.239715 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:13.239269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:39101)
0926 20:13:13.239555 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:13.239615 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260926 20:13:13.426029 6642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:13.425488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:56067)
0926 20:13:13.425861 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:13.425932 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":102,"threads_started":1}
I20260926 20:13:14.084919 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:14.084506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:46553)
0926 20:13:14.084765 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:14.084821 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260926 20:13:14.240900 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:14.240439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:54679)
0926 20:13:14.240731 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:14.240791 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":78,"threads_started":1}
I20260926 20:13:14.427129 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:14.426762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:35735)
0926 20:13:14.426988 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:14.427041 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260926 20:13:14.693580 6646 raft_consensus.cc:482] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:14.693711 6646 raft_consensus.cc:504] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:14.694216 6646 leader_election.cc:288] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309), 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255)
W20260926 20:13:14.695024 6249 leader_election.cc:341] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255): Not found: Tablet not found: 3b56501612cd4b97829ebb2f16fef3bd
W20260926 20:13:14.695106 6250 leader_election.cc:341] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309): Not found: Tablet not found: 3b56501612cd4b97829ebb2f16fef3bd
I20260926 20:13:14.695219 6250 leader_election.cc:302] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82531edff95844979c4610ec2b514f66; no voters: 0b5d71e02915496594f99d7cc851d253, e1a6c17d045c429e9609fdf788430a59
I20260926 20:13:14.695372 6646 raft_consensus.cc:2726] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:13:15.086314 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:15.085822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:59649)
0926 20:13:15.086142 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:15.086214 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":70,"threads_started":1}
I20260926 20:13:15.242440 6648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:15.241902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:47861)
0926 20:13:15.242256 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:15.242324 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
I20260926 20:13:15.428390 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:15.427936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:47277)
0926 20:13:15.428203 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:15.428266 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260926 20:13:16.087715 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:16.087219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:46449)
0926 20:13:16.087560 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:16.087614 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":78,"threads_started":1}
I20260926 20:13:16.243741 6651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:16.243267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:33319)
0926 20:13:16.243578 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:16.243637 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":87,"threads_started":1}
I20260926 20:13:16.429581 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:16.429176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:38445)
0926 20:13:16.429429 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:16.429488 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260926 20:13:17.088999 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:17.088551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:36189)
0926 20:13:17.088834 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:17.088900 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260926 20:13:17.245090 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:17.244656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:39607)
0926 20:13:17.244921 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:17.244988 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260926 20:13:17.430766 6655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:17.430317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:42847)
0926 20:13:17.430604 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:17.430671 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260926 20:13:17.949389 6656 raft_consensus.cc:482] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:17.949537 6656 raft_consensus.cc:504] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:17.950055 6656 leader_election.cc:288] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309), 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255)
W20260926 20:13:17.950947 6249 leader_election.cc:341] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255): Not found: Tablet not found: 3b56501612cd4b97829ebb2f16fef3bd
W20260926 20:13:17.951052 6250 leader_election.cc:341] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309): Not found: Tablet not found: 3b56501612cd4b97829ebb2f16fef3bd
I20260926 20:13:17.951112 6250 leader_election.cc:302] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82531edff95844979c4610ec2b514f66; no voters: 0b5d71e02915496594f99d7cc851d253, e1a6c17d045c429e9609fdf788430a59
I20260926 20:13:17.951254 6656 raft_consensus.cc:2726] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:13:18.090473 6657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:18.090044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:45119)
0926 20:13:18.090311 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:18.090375 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260926 20:13:18.246486 6658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:18.246009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:34137)
0926 20:13:18.246318 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:18.246386 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":72,"threads_started":1}
I20260926 20:13:18.432078 6659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:18.431616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:38291)
0926 20:13:18.431916 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:18.431976 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":84,"threads_started":1}
I20260926 20:13:19.091763 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:19.091360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:57197)
0926 20:13:19.091604 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:19.091663 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":86,"threads_started":1}
I20260926 20:13:19.247836 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:19.247388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:33137)
0926 20:13:19.247671 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:19.247740 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260926 20:13:19.433326 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:19.432862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:38789)
0926 20:13:19.433139 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:19.433203 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260926 20:13:20.093001 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:20.092584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:45959)
0926 20:13:20.092840 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:20.092907 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260926 20:13:20.249181 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:20.248711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:59367)
0926 20:13:20.249003 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:20.249071 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
I20260926 20:13:20.434713 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:20.434262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:36155)
0926 20:13:20.434552 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:20.434618 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260926 20:13:21.094409 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:21.093988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:41687)
0926 20:13:21.094244 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:21.094309 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260926 20:13:21.250638 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:21.250158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:41545)
0926 20:13:21.250463 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:21.250534 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":97,"threads_started":1}
I20260926 20:13:21.435920 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:21.435501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:47089)
0926 20:13:21.435754 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:21.435814 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260926 20:13:21.650715 6669 raft_consensus.cc:482] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:21.650875 6669 raft_consensus.cc:504] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:21.651423 6669 leader_election.cc:288] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309), 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255)
W20260926 20:13:21.652313 6249 leader_election.cc:341] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255): Not found: Tablet not found: 3b56501612cd4b97829ebb2f16fef3bd
W20260926 20:13:21.652565 6250 leader_election.cc:341] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309): Not found: Tablet not found: 3b56501612cd4b97829ebb2f16fef3bd
I20260926 20:13:21.652642 6250 leader_election.cc:302] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82531edff95844979c4610ec2b514f66; no voters: 0b5d71e02915496594f99d7cc851d253, e1a6c17d045c429e9609fdf788430a59
I20260926 20:13:21.652770 6669 raft_consensus.cc:2726] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:13:22.095870 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:22.095299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:41223)
0926 20:13:22.095611 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:22.095687 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":98,"threads_started":1}
I20260926 20:13:22.251942 6671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:22.251511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:59745)
0926 20:13:22.251782 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:22.251846 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260926 20:13:22.437201 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:22.436787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:59053)
0926 20:13:22.437059 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:22.437111 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
I20260926 20:13:23.097229 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:23.096720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:40545)
0926 20:13:23.097010 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:23.097086 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":88,"threads_started":1}
I20260926 20:13:23.253180 6674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:23.252763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:44035)
0926 20:13:23.253020 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:23.253080 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260926 20:13:23.438609 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:23.438079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:54947)
0926 20:13:23.438436 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:23.438512 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":142,"threads_started":1}
I20260926 20:13:24.098457 6676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:24.098051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:47201)
0926 20:13:24.098289 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:24.098346 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260926 20:13:24.254601 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:24.254131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:39787)
0926 20:13:24.254428 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:24.254498 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":80,"threads_started":1}
I20260926 20:13:24.439931 6678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:24.439481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:59051)
0926 20:13:24.439768 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:24.439835 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
I20260926 20:13:24.570286 6679 raft_consensus.cc:482] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:24.570444 6679 raft_consensus.cc:504] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1a6c17d045c429e9609fdf788430a59" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 37309 } } peers { permanent_uuid: "0b5d71e02915496594f99d7cc851d253" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 36255 } } peers { permanent_uuid: "82531edff95844979c4610ec2b514f66" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41785 } }
I20260926 20:13:24.570847 6679 leader_election.cc:288] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309), 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255)
W20260926 20:13:24.571650 6249 leader_election.cc:341] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0b5d71e02915496594f99d7cc851d253 (127.23.50.131:36255): Not found: Tablet not found: 3b56501612cd4b97829ebb2f16fef3bd
W20260926 20:13:24.571882 6250 leader_election.cc:341] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1a6c17d045c429e9609fdf788430a59 (127.23.50.130:37309): Not found: Tablet not found: 3b56501612cd4b97829ebb2f16fef3bd
I20260926 20:13:24.571995 6250 leader_election.cc:302] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82531edff95844979c4610ec2b514f66; no voters: 0b5d71e02915496594f99d7cc851d253, e1a6c17d045c429e9609fdf788430a59
I20260926 20:13:24.572175 6679 raft_consensus.cc:2726] T 3b56501612cd4b97829ebb2f16fef3bd P 82531edff95844979c4610ec2b514f66 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:13:25.099740 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:25.099281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:45091)
0926 20:13:25.099576 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:25.099638 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":98,"threads_started":1}
I20260926 20:13:25.256068 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:25.255587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:50025)
0926 20:13:25.255902 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:25.255968 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
I20260926 20:13:25.441088 6682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:25.440679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:36919)
0926 20:13:25.440936 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:25.440995 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260926 20:13:26.101150 6683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:26.100721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.129:47449)
0926 20:13:26.100992 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:26.101054 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
I20260926 20:13:26.257599 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:26.257101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.130:39497)
0926 20:13:26.257432 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:26.257497 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":120,"threads_started":1}
I20260926 20:13:26.433068 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 6199
I20260926 20:13:26.438891 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 6365
I20260926 20:13:26.442520 6685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:26.442086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:37837 (local address 127.23.50.131:56765)
0926 20:13:26.442365 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:26.442428 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:37837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
I20260926 20:13:26.444566 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 6500
2026-09-26T20:13:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19510 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:13:26 dist-test-slave-7034 krb5kdc[6688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:13:26 dist-test-slave-7034 krb5kdc[6688](info): set up 2 sockets
Sep 26 20:13:26 dist-test-slave-7034 krb5kdc[6688](info): commencing operation
krb5kdc: starting...
W20260926 20:13:27.486532 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:13:27 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:13:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:13:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:13:27.610052 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42405
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:42405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:27.721877 6701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:27.722213 6701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:27.722301 6701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:27.726467 6701 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:13:27.726548 6701 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:13:27.726575 6701 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:13:27.726600 6701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:13:27.726645 6701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:27.726691 6701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:27.726727 6701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:13:27.726763 6701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:13:27.732214 6701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:42405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:27.733368 6701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:27.734588 6701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:27.743286 6707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:27.743278 6709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:27.743252 6706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:27.743543 6701 server_base.cc:1034] running on GCE node
I20260926 20:13:27.744223 6701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:27.745428 6701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:27.746629 6701 hybrid_clock.cc:648] HybridClock initialized: now 1790453607746599 us; error 53 us; skew 500 ppm
Sep 26 20:13:27 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453607, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:27.753315 6701 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:13:27.754397 6701 webserver.cc:460] Webserver started at http://127.23.50.190:39949/ using document root <none> and password file <none>
I20260926 20:13:27.755020 6701 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:27.755117 6701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:27.755365 6701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:13:27.757478 6701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "c6d41a7493584b43a737d96c60ea6ca9"
format_stamp: "Formatted at 2026-09-26 20:13:27 on dist-test-slave-7034"
server_key: "376b3f9b8983fc140d73c43fb6dd738d"
server_key_iv: "8b7818ec3ff565a9dca1d44c4ecdca49"
server_key_version: "encryptionkey@0"
I20260926 20:13:27.758132 6701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "c6d41a7493584b43a737d96c60ea6ca9"
format_stamp: "Formatted at 2026-09-26 20:13:27 on dist-test-slave-7034"
server_key: "376b3f9b8983fc140d73c43fb6dd738d"
server_key_iv: "8b7818ec3ff565a9dca1d44c4ecdca49"
server_key_version: "encryptionkey@0"
I20260926 20:13:27.762002 6701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:13:27.764484 6716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:27.765718 6701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:13:27.765925 6701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "c6d41a7493584b43a737d96c60ea6ca9"
format_stamp: "Formatted at 2026-09-26 20:13:27 on dist-test-slave-7034"
server_key: "376b3f9b8983fc140d73c43fb6dd738d"
server_key_iv: "8b7818ec3ff565a9dca1d44c4ecdca49"
server_key_version: "encryptionkey@0"
I20260926 20:13:27.766059 6701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:27.800962 6701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:27.804824 6701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:27.805073 6701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:27.812117 6701 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:42405
I20260926 20:13:27.812391 6768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:42405 every 8 connection(s)
I20260926 20:13:27.813135 6701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:13:27.815861 6769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:27.816921 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 6701
I20260926 20:13:27.817065 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:13:27.817384 23754 external_mini_cluster.cc:1428] Setting key 1d4115b1a3a9d63e2759ee159cf759a7
I20260926 20:13:27.822566 6769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Bootstrap starting.
I20260926 20:13:27.824911 6769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:27.825698 6769 log.cc:824] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:13:27 dist-test-slave-7034 krb5kdc[6688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:27.827943 6769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: No bootstrap required, opened a new log
I20260926 20:13:27.831670 6769 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } }
I20260926 20:13:27.831933 6769 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:27.832016 6769 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d41a7493584b43a737d96c60ea6ca9, State: Initialized, Role: FOLLOWER
I20260926 20:13:27.832453 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:27.819005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40622 (local address 127.23.50.190:42405)
0926 20:13:27.819538 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:27.819553 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:27.819591 (+ 38us) server_negotiation.cc:407] Connection header received
0926 20:13:27.820261 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:27.820280 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:27.820652 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:27.821008 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:27.821915 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:27.823002 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:27.823887 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:27.824385 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:27.824568 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:27.827463 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:27.827469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:27.827485 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:27.827514 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:27.829319 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:27.829808 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:27.829814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:27.829819 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:27.829993 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:27.830282 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:27.830285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:27.830287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:27.830702 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:27.830938 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:27.831201 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 20:13:27.831516 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":141,"threads_started":1}
I20260926 20:13:27.832482 6769 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } }
I20260926 20:13:27.832710 6769 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:13:27.832787 6769 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:13:27.832887 6769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:27.834085 6769 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } }
I20260926 20:13:27.834496 6769 leader_election.cc:302] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6d41a7493584b43a737d96c60ea6ca9; no voters:
I20260926 20:13:27.835024 6769 leader_election.cc:288] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:13:27.835127 6774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:13:27.835390 6774 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 1 LEADER]: Becoming Leader. State: Replica: c6d41a7493584b43a737d96c60ea6ca9, State: Running, Role: LEADER
I20260926 20:13:27.835716 6774 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } }
I20260926 20:13:27.836401 6769 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:13:27.837532 6775 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6d41a7493584b43a737d96c60ea6ca9. Latest consensus state: current_term: 1 leader_uuid: "c6d41a7493584b43a737d96c60ea6ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } } }
I20260926 20:13:27.837539 6776 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6d41a7493584b43a737d96c60ea6ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } } }
I20260926 20:13:27.837800 6775 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:27.837828 6776 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:27.838635 6783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:13:27.841511 6783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:13:27.847484 6783 catalog_manager.cc:1347] Generated new cluster ID: 8a2581fa9bc1487689332390b9d01a52
I20260926 20:13:27.847560 6783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:13:27.874442 6783 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:13:27.875747 6783 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:13:27.891105 6783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Generated new TSK 0
I20260926 20:13:27.892066 6783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:13:27.920483 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42405
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:28.034870 6795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:28.035122 6795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:28.035164 6795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:28.039536 6795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:28.039592 6795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:28.039671 6795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:13:28.044847 6795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:28.045970 6795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:28.046989 6795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:28.056612 6801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:28.056612 6800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:28.056819 6795 server_base.cc:1034] running on GCE node
W20260926 20:13:28.056618 6803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:28.057530 6795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:28.058197 6795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:28.059388 6795 hybrid_clock.cc:648] HybridClock initialized: now 1790453608059356 us; error 45 us; skew 500 ppm
Sep 26 20:13:28 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453608, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:28.062142 6795 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:13:28.063012 6795 webserver.cc:460] Webserver started at http://127.23.50.129:32771/ using document root <none> and password file <none>
I20260926 20:13:28.063582 6795 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:28.063675 6795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:28.063910 6795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:13:28.065970 6795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "01868c6c2ec4413d87294a208538e121"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "84dac65d01fa17d8bd319aa547640d34"
server_key_iv: "fd0b63ab5cffb5dc6d1aab0875cfc6a6"
server_key_version: "encryptionkey@0"
I20260926 20:13:28.066548 6795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "01868c6c2ec4413d87294a208538e121"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "84dac65d01fa17d8bd319aa547640d34"
server_key_iv: "fd0b63ab5cffb5dc6d1aab0875cfc6a6"
server_key_version: "encryptionkey@0"
I20260926 20:13:28.070358 6795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:13:28.072839 6810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:28.074290 6795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:13:28.074461 6795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "01868c6c2ec4413d87294a208538e121"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "84dac65d01fa17d8bd319aa547640d34"
server_key_iv: "fd0b63ab5cffb5dc6d1aab0875cfc6a6"
server_key_version: "encryptionkey@0"
I20260926 20:13:28.074589 6795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:28.090461 6795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:28.093743 6795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:28.093966 6795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:28.094521 6795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:28.095294 6795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:13:28.095340 6795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:28.095381 6795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:28.095396 6795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:28.104470 6795 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:45491
I20260926 20:13:28.104499 6923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:45491 every 8 connection(s)
I20260926 20:13:28.105388 6795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:13:28.106928 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 6795
I20260926 20:13:28.107069 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:13:28.107429 23754 external_mini_cluster.cc:1428] Setting key aef0ec772bd03df2971bb08f6d4e271e
Sep 26 20:13:28 dist-test-slave-7034 krb5kdc[6688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453608, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
I20260926 20:13:28.121268 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.107315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:59075 (local address 127.23.50.190:42405)
0926 20:13:28.107487 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:28.107493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:28.108530 (+ 1037us) server_negotiation.cc:407] Connection header received
0926 20:13:28.110501 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:28.110503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:28.110550 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:28.110626 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:28.111923 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.112826 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.114126 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.114588 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.114727 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.117227 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:28.117234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:28.117239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:28.117275 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:28.118806 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:28.119425 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:28.119429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:28.119430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:28.119507 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:28.119952 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:28.119955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:28.119957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:28.120180 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:28.120296 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:28.120842 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 20:13:28.121011 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:13:28.122042 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.107830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:59075)
0926 20:13:28.108384 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:28.108430 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:28.110246 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:28.110811 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:28.110822 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:28.111282 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:28.111768 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.111782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.112953 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.112956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.114001 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.114011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.114747 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.114750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.114900 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.115538 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:28.115556 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:28.117029 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:28.118923 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:28.118929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:28.118964 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:28.119330 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:28.119613 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:28.119616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:28.119620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:28.119843 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:28.120319 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:28.120324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:28.120577 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 20:13:28.120748 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 20:13:28.121027 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":156,"threads_started":1}
I20260926 20:13:28.123397 6924 heartbeater.cc:347] Connected to a master server at 127.23.50.190:42405
I20260926 20:13:28.123670 6924 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:28.124287 6924 heartbeater.cc:511] Master 127.23.50.190:42405 requested a full tablet report, sending...
I20260926 20:13:28.126117 6733 ts_manager.cc:194] Registered new tserver with Master: 01868c6c2ec4413d87294a208538e121 (127.23.50.129:45491)
I20260926 20:13:28.127323 6733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:59075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:13:28.138245 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42405
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:28.252883 6929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:28.253264 6929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:28.253347 6929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:28.257579 6929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:28.257685 6929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:28.257789 6929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:13:28.263476 6929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:28.264647 6929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:28.265753 6929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:28.275983 6937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:28.276132 6929 server_base.cc:1034] running on GCE node
W20260926 20:13:28.275983 6934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:28.276002 6935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:28.276732 6929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:28.277343 6929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:28.278514 6929 hybrid_clock.cc:648] HybridClock initialized: now 1790453608278488 us; error 54 us; skew 500 ppm
Sep 26 20:13:28 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453608, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:28.281353 6929 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:13:28.282260 6929 webserver.cc:460] Webserver started at http://127.23.50.130:35335/ using document root <none> and password file <none>
I20260926 20:13:28.282787 6929 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:28.282846 6929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:28.283010 6929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:13:28.285059 6929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "3ab5b58d10b4417c91f6271bc3a3d452"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "12db4c52cdbbb13bfeb642638465e1b7"
server_key_iv: "7168c70e3a84689ba27fc7586a47ae5b"
server_key_version: "encryptionkey@0"
I20260926 20:13:28.285578 6929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "3ab5b58d10b4417c91f6271bc3a3d452"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "12db4c52cdbbb13bfeb642638465e1b7"
server_key_iv: "7168c70e3a84689ba27fc7586a47ae5b"
server_key_version: "encryptionkey@0"
I20260926 20:13:28.289299 6929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:13:28.291759 6944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:28.293004 6929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:13:28.293129 6929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "3ab5b58d10b4417c91f6271bc3a3d452"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "12db4c52cdbbb13bfeb642638465e1b7"
server_key_iv: "7168c70e3a84689ba27fc7586a47ae5b"
server_key_version: "encryptionkey@0"
I20260926 20:13:28.293238 6929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:28.323668 6929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:28.327093 6929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:28.327313 6929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:28.327940 6929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:28.328815 6929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:13:28.328863 6929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:28.328934 6929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:28.328971 6929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:28.338105 6929 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:39265
I20260926 20:13:28.338122 7057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:39265 every 8 connection(s)
I20260926 20:13:28.339260 6929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:13:28.345130 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 6929
I20260926 20:13:28.345274 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:13:28.345548 23754 external_mini_cluster.cc:1428] Setting key 38f16678e7919b11d49c6849ae4fcb9d
Sep 26 20:13:28 dist-test-slave-7034 krb5kdc[6688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453608, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:28.355266 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.341168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:55059 (local address 127.23.50.190:42405)
0926 20:13:28.341312 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:28.341315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:28.342500 (+ 1185us) server_negotiation.cc:407] Connection header received
0926 20:13:28.343919 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:28.343922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:28.343981 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:28.344085 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:28.345307 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.346045 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.347795 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.348013 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.348114 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.350835 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:28.350842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:28.350848 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:28.350884 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:28.352680 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:28.353492 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:28.353499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:28.353503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:28.353602 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:28.354006 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:28.354011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:28.354013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:28.354277 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:28.354390 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:28.354901 (+ 511us) server_negotiation.cc:299] Negotiation successful
0926 20:13:28.355036 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:13:28.356065 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.341572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:55059)
0926 20:13:28.342317 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:28.342364 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:28.343687 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:28.344272 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:28.344280 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:28.344651 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:28.345156 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.345170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.346248 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.346252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.347594 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.347608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.348193 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.348200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.348447 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.349113 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:28.349130 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:28.350653 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:28.352955 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:28.352960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:28.352973 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:28.353383 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:28.353701 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:28.353708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:28.353709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:28.353912 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:28.354389 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:28.354394 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:28.354638 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 20:13:28.354798 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 20:13:28.355062 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":124,"threads_started":1}
I20260926 20:13:28.357309 7058 heartbeater.cc:347] Connected to a master server at 127.23.50.190:42405
I20260926 20:13:28.357549 7058 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:28.358114 7058 heartbeater.cc:511] Master 127.23.50.190:42405 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
I20260926 20:13:28.359262 6733 ts_manager.cc:194] Registered new tserver with Master: 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265)
I20260926 20:13:28.360157 6733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:55059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:13:28.373041 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42405
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:28.485308 7063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:28.485625 7063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:28.485720 7063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:28.490024 7063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:28.490100 7063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:28.490197 7063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:13:28.495409 7063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:28.496610 7063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:28.497738 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:28.507182 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:28.507189 7068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:28.507445 7063 server_base.cc:1034] running on GCE node
W20260926 20:13:28.507213 7069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:28.507826 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:28.508534 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:28.509719 7063 hybrid_clock.cc:648] HybridClock initialized: now 1790453608509701 us; error 27 us; skew 500 ppm
Sep 26 20:13:28 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453608, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:28.512478 7063 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:13:28.513300 7063 webserver.cc:460] Webserver started at http://127.23.50.131:43241/ using document root <none> and password file <none>
I20260926 20:13:28.513825 7063 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:28.513957 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:28.514153 7063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:13:28.516131 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "e2671a5362ec44c69479de3428f9977a"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "f0368eb60fbe90844729e7e92200442d"
server_key_iv: "5600945550f4bd1fbe35f0a2cf4057b6"
server_key_version: "encryptionkey@0"
I20260926 20:13:28.516635 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "e2671a5362ec44c69479de3428f9977a"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "f0368eb60fbe90844729e7e92200442d"
server_key_iv: "5600945550f4bd1fbe35f0a2cf4057b6"
server_key_version: "encryptionkey@0"
I20260926 20:13:28.520319 7063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 20:13:28.522599 7078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:28.523841 7063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:13:28.523973 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "e2671a5362ec44c69479de3428f9977a"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "f0368eb60fbe90844729e7e92200442d"
server_key_iv: "5600945550f4bd1fbe35f0a2cf4057b6"
server_key_version: "encryptionkey@0"
I20260926 20:13:28.524085 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:28.552225 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:28.555461 7063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:28.555658 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:28.556268 7063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:28.557114 7063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:13:28.557186 7063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:28.557260 7063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:28.557315 7063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:28.566314 7063 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:34853
I20260926 20:13:28.566339 7191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:34853 every 8 connection(s)
I20260926 20:13:28.567322 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:13:28.569593 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 7063
I20260926 20:13:28.569726 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:13:28.570070 23754 external_mini_cluster.cc:1428] Setting key da1ca49c2594baae6d03cdc3082a6e07
Sep 26 20:13:28 dist-test-slave-7034 krb5kdc[6688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453608, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:28.582589 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.569430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:60671 (local address 127.23.50.190:42405)
0926 20:13:28.569592 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:28.569597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:28.570834 (+ 1237us) server_negotiation.cc:407] Connection header received
0926 20:13:28.572502 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:28.572506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:28.572562 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:28.572636 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:28.573943 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.574496 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.575722 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.576038 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.576157 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.578798 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:28.578802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:28.578806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:28.578845 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:28.580316 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:28.580877 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:28.580881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:28.580883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:28.580971 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:28.581344 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:28.581348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:28.581350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:28.581551 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:28.581657 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:28.582281 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 20:13:28.582422 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:13:28.583653 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.569973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:60671)
0926 20:13:28.570667 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:28.570728 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:28.572244 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:28.572833 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:28.572841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:28.573229 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:28.573731 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.573745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.574634 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.574638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.575557 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.575567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.576202 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.576207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.576393 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.576971 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:28.576987 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:28.578617 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:28.580454 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:28.580461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:28.580478 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:28.580760 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:28.581075 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:28.581078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:28.581080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:28.581232 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:28.581670 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:28.581679 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:28.582028 (+ 349us) client_negotiation.cc:769] Sending connection context
0926 20:13:28.582225 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 20:13:28.582587 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":155,"threads_started":1}
I20260926 20:13:28.584807 7192 heartbeater.cc:347] Connected to a master server at 127.23.50.190:42405
I20260926 20:13:28.585083 7192 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:28.585585 7192 heartbeater.cc:511] Master 127.23.50.190:42405 requested a full tablet report, sending...
I20260926 20:13:28.586745 6732 ts_manager.cc:194] Registered new tserver with Master: e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853)
I20260926 20:13:28.587335 6732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:60671
I20260926 20:13:28.596495 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:13:28.605676 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.598603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40626 (local address 127.23.50.190:42405)
0926 20:13:28.598733 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:28.598737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:28.598868 (+ 131us) server_negotiation.cc:407] Connection header received
0926 20:13:28.598976 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:28.598979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:28.599022 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:28.599083 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:28.599768 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.600281 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.601085 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.601298 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.601376 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.602740 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:28.602743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:28.602745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:28.602780 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:28.604223 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:28.604627 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:28.604629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:28.604630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:28.604706 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:28.604929 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:28.604931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:28.604932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:28.605145 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:28.605248 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:28.605460 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 20:13:28.605557 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:13:28.609212 6732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:13:28.612460 6732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:13:28.627283 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.621649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.23.50.130:39265)
0926 20:13:28.622608 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:28.622622 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:28.622647 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:13:28.622720 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:28.622725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:28.622915 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:28.623370 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:28.623521 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.624586 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.625921 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.626510 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.626639 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.626802 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:28.627013 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 20:13:28.627117 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":180,"threads_started":1}
I20260926 20:13:28.627271 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.620997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:39265 (local address 127.0.0.1:54062)
0926 20:13:28.621439 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:28.621454 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:28.621551 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:28.623095 (+ 1544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:28.623098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:28.623116 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:28.623214 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.623221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.624729 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.624733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.625763 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.625773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.626637 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.626640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.626766 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.626807 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 20:13:28.626947 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 20:13:28.627105 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":26,"thread_start_us":118,"threads_started":1}
I20260926 20:13:28.627573 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.621740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.23.50.131:34853)
0926 20:13:28.622176 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:28.622190 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:28.622217 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:13:28.622302 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:28.622309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:28.622541 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:28.622706 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:28.623014 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.624194 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.625938 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.626691 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.626787 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.626931 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:28.627345 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 20:13:28.627429 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":155,"threads_started":1}
I20260926 20:13:28.627573 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.622162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.23.50.129:45491)
0926 20:13:28.622534 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:28.622549 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:28.622576 (+ 27us) server_negotiation.cc:407] Connection header received
0926 20:13:28.622653 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:28.622660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:28.622873 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:28.623058 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:28.623229 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.623952 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.625749 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.626623 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.627131 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.627287 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:28.627340 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 20:13:28.627429 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
I20260926 20:13:28.627867 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.621715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:45491 (local address 127.0.0.1:50770)
0926 20:13:28.622200 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:28.622215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:28.622304 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:28.623027 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:28.623030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:28.623047 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:28.623136 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.623142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.624068 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.624073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.625616 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.625627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.626775 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.626778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.626886 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.626910 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:13:28.626965 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 20:13:28.627692 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":695,"thread_start_us":97,"threads_started":1}
I20260926 20:13:28.627873 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.620922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:34853 (local address 127.0.0.1:33244)
0926 20:13:28.621274 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:28.621322 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:28.621443 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:28.622705 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:28.622711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:28.622735 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:28.622877 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.622886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.624336 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.624339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.625616 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.625627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.626992 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.626995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.627085 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.627110 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:13:28.627173 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 20:13:28.627242 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":70,"thread_start_us":94,"threads_started":1}
I20260926 20:13:28.630824 6992 tablet_service.cc:1467] Processing CreateTablet for tablet e1bc3cf4d25f453897b49e09063421ba (DEFAULT_TABLE table=test-table [id=2477da5de1994a8991249859214d2dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:28.630863 6858 tablet_service.cc:1467] Processing CreateTablet for tablet e1bc3cf4d25f453897b49e09063421ba (DEFAULT_TABLE table=test-table [id=2477da5de1994a8991249859214d2dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:28.630865 7126 tablet_service.cc:1467] Processing CreateTablet for tablet e1bc3cf4d25f453897b49e09063421ba (DEFAULT_TABLE table=test-table [id=2477da5de1994a8991249859214d2dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:28.631852 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1bc3cf4d25f453897b49e09063421ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:28.631870 6858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1bc3cf4d25f453897b49e09063421ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:28.631877 7126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1bc3cf4d25f453897b49e09063421ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:28.638774 7209 tablet_bootstrap.cc:492] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: Bootstrap starting.
I20260926 20:13:28.639930 7210 tablet_bootstrap.cc:492] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a: Bootstrap starting.
I20260926 20:13:28.640129 7211 tablet_bootstrap.cc:492] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Bootstrap starting.
I20260926 20:13:28.641642 7210 tablet_bootstrap.cc:654] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:28.641644 7209 tablet_bootstrap.cc:654] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:28.642277 7211 tablet_bootstrap.cc:654] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:28.642613 7209 log.cc:824] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:28.642849 7210 log.cc:824] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:28.643071 7211 log.cc:824] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:28.644973 7210 tablet_bootstrap.cc:492] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a: No bootstrap required, opened a new log
I20260926 20:13:28.644973 7211 tablet_bootstrap.cc:492] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: No bootstrap required, opened a new log
I20260926 20:13:28.644996 7209 tablet_bootstrap.cc:492] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: No bootstrap required, opened a new log
I20260926 20:13:28.645135 7209 ts_tablet_manager.cc:1397] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260926 20:13:28.645154 7211 ts_tablet_manager.cc:1397] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:13:28.645196 7210 ts_tablet_manager.cc:1397] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260926 20:13:28.647980 7209 raft_consensus.cc:348] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:28.648250 7209 raft_consensus.cc:374] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:28.648324 7209 raft_consensus.cc:729] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab5b58d10b4417c91f6271bc3a3d452, State: Initialized, Role: FOLLOWER
I20260926 20:13:28.648779 7209 consensus_queue.cc:260] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:28.649080 7210 raft_consensus.cc:348] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:28.649407 7210 raft_consensus.cc:374] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:28.649464 7058 heartbeater.cc:503] Master 127.23.50.190:42405 was elected leader, sending a full tablet report...
I20260926 20:13:28.649513 7210 raft_consensus.cc:729] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2671a5362ec44c69479de3428f9977a, State: Initialized, Role: FOLLOWER
I20260926 20:13:28.649458 7211 raft_consensus.cc:348] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:28.649691 7211 raft_consensus.cc:374] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:28.649741 7211 raft_consensus.cc:729] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01868c6c2ec4413d87294a208538e121, State: Initialized, Role: FOLLOWER
I20260926 20:13:28.649784 7209 ts_tablet_manager.cc:1428] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260926 20:13:28.649995 7210 consensus_queue.cc:260] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:28.650146 7211 consensus_queue.cc:260] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:28.650635 7192 heartbeater.cc:503] Master 127.23.50.190:42405 was elected leader, sending a full tablet report...
I20260926 20:13:28.650929 7210 ts_tablet_manager.cc:1428] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:13:28.650929 7211 ts_tablet_manager.cc:1428] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:13:28.651629 6924 heartbeater.cc:503] Master 127.23.50.190:42405 was elected leader, sending a full tablet report...
I20260926 20:13:28.656404 7217 raft_consensus.cc:482] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:28.656562 7217 raft_consensus.cc:504] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:28.657573 7217 leader_election.cc:288] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853), 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265)
I20260926 20:13:28.661520 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.657828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:55087 (local address 127.23.50.131:34853)
0926 20:13:28.658105 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:28.658109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:28.658172 (+ 63us) server_negotiation.cc:407] Connection header received
0926 20:13:28.658274 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:28.658277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:28.658372 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:28.658449 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:28.658749 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.659335 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.660509 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.660909 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.661037 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.661093 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:28.661330 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 20:13:28.661395 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260926 20:13:28.661525 7206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.657955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:43473 (local address 127.23.50.130:39265)
0926 20:13:28.658068 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:28.658074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:28.658397 (+ 323us) server_negotiation.cc:407] Connection header received
0926 20:13:28.658492 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:28.658495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:28.658539 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:28.658616 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:28.658907 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.659399 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.660504 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.660911 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.661036 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.661089 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:28.661348 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 20:13:28.661414 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:13:28.661551 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.657772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:34853 (local address 127.23.50.129:55087)
0926 20:13:28.658068 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:28.658084 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:28.658197 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:28.658471 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:28.658474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:28.658493 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:28.658639 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.658646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.659447 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.659453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.660384 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.660393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.661030 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.661033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.661188 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.661208 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 20:13:28.661269 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 20:13:28.661360 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":56,"thread_start_us":134,"threads_started":1}
I20260926 20:13:28.661554 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.657774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:39265 (local address 127.23.50.129:43473)
0926 20:13:28.658294 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:28.658308 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:28.658413 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:28.658633 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:28.658635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:28.658648 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:28.658789 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.658795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.659519 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.659522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.660382 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:28.660391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.661029 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.661033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:28.661207 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.661224 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 20:13:28.661319 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:13:28.661389 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":114,"thread_start_us":69,"threads_started":1}
I20260926 20:13:28.662189 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1bc3cf4d25f453897b49e09063421ba" candidate_uuid: "01868c6c2ec4413d87294a208538e121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" is_pre_election: true
I20260926 20:13:28.662498 7012 raft_consensus.cc:2454] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01868c6c2ec4413d87294a208538e121 in term 0.
I20260926 20:13:28.662631 7146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1bc3cf4d25f453897b49e09063421ba" candidate_uuid: "01868c6c2ec4413d87294a208538e121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2671a5362ec44c69479de3428f9977a" is_pre_election: true
I20260926 20:13:28.662933 7146 raft_consensus.cc:2454] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01868c6c2ec4413d87294a208538e121 in term 0.
I20260926 20:13:28.663149 6812 leader_election.cc:302] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01868c6c2ec4413d87294a208538e121, 3ab5b58d10b4417c91f6271bc3a3d452; no voters:
I20260926 20:13:28.663422 7217 raft_consensus.cc:2781] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:13:28.663488 7217 raft_consensus.cc:482] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:13:28.663517 7217 raft_consensus.cc:3037] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:28.664484 7217 raft_consensus.cc:504] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:28.664785 7217 leader_election.cc:288] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 1 election: Requested vote from peers e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853), 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265)
I20260926 20:13:28.665122 7146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1bc3cf4d25f453897b49e09063421ba" candidate_uuid: "01868c6c2ec4413d87294a208538e121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2671a5362ec44c69479de3428f9977a"
I20260926 20:13:28.665162 7012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1bc3cf4d25f453897b49e09063421ba" candidate_uuid: "01868c6c2ec4413d87294a208538e121" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab5b58d10b4417c91f6271bc3a3d452"
I20260926 20:13:28.665287 7146 raft_consensus.cc:3037] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:28.665294 7012 raft_consensus.cc:3037] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:28.666388 7012 raft_consensus.cc:2454] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01868c6c2ec4413d87294a208538e121 in term 1.
I20260926 20:13:28.666463 7146 raft_consensus.cc:2454] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01868c6c2ec4413d87294a208538e121 in term 1.
I20260926 20:13:28.666703 6812 leader_election.cc:302] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01868c6c2ec4413d87294a208538e121, 3ab5b58d10b4417c91f6271bc3a3d452; no voters:
I20260926 20:13:28.666921 7217 raft_consensus.cc:2781] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:13:28.667140 7217 raft_consensus.cc:686] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 1 LEADER]: Becoming Leader. State: Replica: 01868c6c2ec4413d87294a208538e121, State: Running, Role: LEADER
I20260926 20:13:28.667456 7217 consensus_queue.cc:237] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:28.670580 6731 catalog_manager.cc:5495] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01868c6c2ec4413d87294a208538e121 (127.23.50.129). New cstate: current_term: 1 leader_uuid: "01868c6c2ec4413d87294a208538e121" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } health_report { overall_health: HEALTHY } } }
I20260926 20:13:28.680639 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.677264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.23.50.129:45491)
0926 20:13:28.677390 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:28.677394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:28.677407 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:13:28.677558 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:28.677560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:28.677609 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:28.677705 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:13:28.677981 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.678510 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.679375 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:28.679578 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:28.679655 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:28.679895 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:13:28.680284 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:13:28.680371 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:28.680424 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 20:13:28.680471 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:13:28.686494 7146 raft_consensus.cc:1261] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a [term 1 FOLLOWER]: Refusing update from remote peer 01868c6c2ec4413d87294a208538e121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:13:28.686494 7012 raft_consensus.cc:1261] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452 [term 1 FOLLOWER]: Refusing update from remote peer 01868c6c2ec4413d87294a208538e121: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:13:28.687178 7220 consensus_queue.cc:1035] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:13:28.687371 7217 consensus_queue.cc:1035] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:13:28.694670 7223 mvcc.cc:205] Tried to move back new op lower bound from 7333697981172092928 to 7333697981104615424. Current Snapshot: MvccSnapshot[applied={T|T < 7333697981172092928}]
I20260926 20:13:28.698437 7225 mvcc.cc:205] Tried to move back new op lower bound from 7333697981172092928 to 7333697981104615424. Current Snapshot: MvccSnapshot[applied={T|T < 7333697981172092928}]
I20260926 20:13:28.698508 7224 mvcc.cc:205] Tried to move back new op lower bound from 7333697981172092928 to 7333697981104615424. Current Snapshot: MvccSnapshot[applied={T|T < 7333697981172092928 or (T in {7333697981172092928})}]
I20260926 20:13:28.703497 6731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:13:28.703677 6731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:13:28.706203 6731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Sending DeleteTablet for 3 replicas of tablet e1bc3cf4d25f453897b49e09063421ba
I20260926 20:13:28.707058 7126 tablet_service.cc:1514] Processing DeleteTablet for tablet e1bc3cf4d25f453897b49e09063421ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:28 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:33244
I20260926 20:13:28.707099 6992 tablet_service.cc:1514] Processing DeleteTablet for tablet e1bc3cf4d25f453897b49e09063421ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:28 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:54062
I20260926 20:13:28.707188 6858 tablet_service.cc:1514] Processing DeleteTablet for tablet e1bc3cf4d25f453897b49e09063421ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:28 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:50770
I20260926 20:13:28.707597 7233 tablet_replica.cc:330] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a: stopping tablet replica
I20260926 20:13:28.707639 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 6795
I20260926 20:13:28.707950 7233 raft_consensus.cc:2229] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:13:28.708578 7233 raft_consensus.cc:2258] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:28.710212 7233 ts_tablet_manager.cc:1905] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:28.712507 7234 tablet_replica.cc:330] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: stopping tablet replica
I20260926 20:13:28.712862 7234 raft_consensus.cc:2229] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:13:28.713207 7234 raft_consensus.cc:2258] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:28.713495 7233 ts_tablet_manager.cc:1918] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:13:28.713637 7233 log.cc:1196] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/e1bc3cf4d25f453897b49e09063421ba
I20260926 20:13:28.714048 7233 ts_tablet_manager.cc:1939] T e1bc3cf4d25f453897b49e09063421ba P e2671a5362ec44c69479de3428f9977a: Deleting consensus metadata
I20260926 20:13:28.714915 7234 ts_tablet_manager.cc:1905] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:28.715444 6720 catalog_manager.cc:4841] TS e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853): tablet e1bc3cf4d25f453897b49e09063421ba (table test-table [id=2477da5de1994a8991249859214d2dce]) successfully deleted
I20260926 20:13:28.717316 7234 ts_tablet_manager.cc:1918] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:13:28.717434 7234 log.cc:1196] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/e1bc3cf4d25f453897b49e09063421ba
W20260926 20:13:28.717545 6720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:45491 (error 108)
I20260926 20:13:28.717806 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 6929
I20260926 20:13:28.717837 7234 ts_tablet_manager.cc:1939] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: Deleting consensus metadata
I20260926 20:13:28.720471 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.720153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:45491 (local address 127.0.0.1:50792)
0926 20:13:28.720321 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:28.720392 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 20:13:28.720651 6720 catalog_manager.cc:4584] TS 01868c6c2ec4413d87294a208538e121 (127.23.50.129:45491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1bc3cf4d25f453897b49e09063421ba: Network error: Client connection negotiation failed: client connection to 127.23.50.129:45491: connect: Connection refused (error 111)
I20260926 20:13:28.724658 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 7063
I20260926 20:13:28.726487 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:28.724872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:39265 (local address 127.0.0.1:54064)
0926 20:13:28.726300 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:28.726375 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.130:39265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1307}
I20260926 20:13:28.730703 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 6701
I20260926 20:13:48.737682 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42405
--webserver_interface=127.23.50.190
--webserver_port=39949
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:42405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:13:48.851040 7237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:48.851377 7237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:48.851464 7237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:48.855597 7237 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:13:48.855683 7237 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:13:48.855710 7237 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:13:48.855748 7237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:13:48.855791 7237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:48.855852 7237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:48.855890 7237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:13:48.855924 7237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:13:48.861572 7237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:42405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=39949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:48.863070 7237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:48.864315 7237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:48.878239 7243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:48.878224 7244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:48.878221 7246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:48.878299 7237 server_base.cc:1034] running on GCE node
I20260926 20:13:48.879159 7237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:48.880364 7237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:48.881577 7237 hybrid_clock.cc:648] HybridClock initialized: now 1790453628881546 us; error 54 us; skew 500 ppm
Sep 26 20:13:48 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453628, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:48.890138 7237 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:13:48.891181 7237 webserver.cc:460] Webserver started at http://127.23.50.190:39949/ using document root <none> and password file <none>
I20260926 20:13:48.891819 7237 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:48.891932 7237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:48.896107 7237 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:13:48.898517 7253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:48.899884 7237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:13:48.900049 7237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "c6d41a7493584b43a737d96c60ea6ca9"
format_stamp: "Formatted at 2026-09-26 20:13:27 on dist-test-slave-7034"
server_key: "376b3f9b8983fc140d73c43fb6dd738d"
server_key_iv: "8b7818ec3ff565a9dca1d44c4ecdca49"
server_key_version: "encryptionkey@0"
I20260926 20:13:48.900693 7237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:48.920240 7237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:48.924290 7237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:48.924561 7237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:48.932694 7237 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:42405
I20260926 20:13:48.933001 7305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:42405 every 8 connection(s)
I20260926 20:13:48.933988 7237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:13:48.935194 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 7237
I20260926 20:13:48.935781 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:45491
--local_ip_for_outbound_sockets=127.23.50.129
--tserver_master_addrs=127.23.50.190:42405
--webserver_port=32771
--webserver_interface=127.23.50.129
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:13:48.939844 7306 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:13:48.941695 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Bootstrap starting.
I20260926 20:13:48.959810 7306 log.cc:824] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:48.968887 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:13:48.969329 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Bootstrap complete.
I20260926 20:13:48.973574 7306 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } }
I20260926 20:13:48.974084 7306 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d41a7493584b43a737d96c60ea6ca9, State: Initialized, Role: FOLLOWER
I20260926 20:13:48.974653 7306 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } }
I20260926 20:13:48.974804 7306 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:13:48.974910 7306 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:13:48.975026 7306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 1 FOLLOWER]: Advancing to term 2
I20260926 20:13:48.977007 7306 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } }
I20260926 20:13:48.977371 7306 leader_election.cc:302] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6d41a7493584b43a737d96c60ea6ca9; no voters:
I20260926 20:13:48.977703 7306 leader_election.cc:288] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 20:13:48.977908 7310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 20:13:48.978274 7310 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 2 LEADER]: Becoming Leader. State: Replica: c6d41a7493584b43a737d96c60ea6ca9, State: Running, Role: LEADER
I20260926 20:13:48.978673 7310 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } }
I20260926 20:13:48.979003 7306 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:13:48.980219 7311 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c6d41a7493584b43a737d96c60ea6ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } } }
I20260926 20:13:48.980351 7311 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:48.981106 7312 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6d41a7493584b43a737d96c60ea6ca9. Latest consensus state: current_term: 2 leader_uuid: "c6d41a7493584b43a737d96c60ea6ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } } }
I20260926 20:13:48.981591 7312 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:13:48.983929 7327 catalog_manager.cc:1259] Loaded cluster ID: 8a2581fa9bc1487689332390b9d01a52
I20260926 20:13:48.984009 7327 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: loading cluster ID for follower catalog manager: success
I20260926 20:13:48.985811 7327 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: acquiring CA information for follower catalog manager: success
I20260926 20:13:48.987103 7327 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:13:48.987260 7320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:13:48.989250 7320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:13:48.989722 7320 catalog_manager.cc:1259] Loaded cluster ID: 8a2581fa9bc1487689332390b9d01a52
I20260926 20:13:48.989799 7320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:13:48.990437 7320 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:13:48.990895 7320 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Loaded TSK: 0
I20260926 20:13:49.004146 7320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Generated new TSK 1
I20260926 20:13:49.004473 7320 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:13:49.072377 7308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:49.072654 7308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:49.072731 7308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:49.076892 7308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:49.076958 7308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:49.077045 7308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:13:49.082767 7308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:45491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=32771
--tserver_master_addrs=127.23.50.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:49.083861 7308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:49.084913 7308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:49.095841 7335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:49.095841 7334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:49.095943 7337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:49.096228 7308 server_base.cc:1034] running on GCE node
I20260926 20:13:49.096750 7308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:49.097566 7308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:49.098744 7308 hybrid_clock.cc:648] HybridClock initialized: now 1790453629098707 us; error 58 us; skew 500 ppm
Sep 26 20:13:49 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453629, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:49.101524 7308 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:13:49.102576 7308 webserver.cc:460] Webserver started at http://127.23.50.129:32771/ using document root <none> and password file <none>
I20260926 20:13:49.103138 7308 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:49.103220 7308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:49.107376 7308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:13:49.109642 7344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:49.111068 7308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:13:49.111196 7308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "01868c6c2ec4413d87294a208538e121"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "84dac65d01fa17d8bd319aa547640d34"
server_key_iv: "fd0b63ab5cffb5dc6d1aab0875cfc6a6"
server_key_version: "encryptionkey@0"
I20260926 20:13:49.111759 7308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:49.139737 7308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:49.142988 7308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:49.143221 7308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:49.143898 7308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:49.145051 7351 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:13:49.149179 7308 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:13:49.149250 7308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 20:13:49.149322 7308 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:13:49.151110 7308 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:13:49.151180 7308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260926 20:13:49.151427 7351 tablet_bootstrap.cc:492] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Bootstrap starting.
I20260926 20:13:49.161065 7308 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:45491
I20260926 20:13:49.161329 7458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:45491 every 8 connection(s)
I20260926 20:13:49.162213 7308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:13:49.163470 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 7308
I20260926 20:13:49.164008 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:39265
--local_ip_for_outbound_sockets=127.23.50.130
--tserver_master_addrs=127.23.50.190:42405
--webserver_port=35335
--webserver_interface=127.23.50.130
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:13:49.173257 7351 log.cc:824] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:13:49 dist-test-slave-7034 krb5kdc[6688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453629, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:49.182374 7351 tablet_bootstrap.cc:492] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:13:49.182886 7351 tablet_bootstrap.cc:492] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Bootstrap complete.
I20260926 20:13:49.183163 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.164598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:60479)
0926 20:13:49.165239 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:49.165302 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:49.166510 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:49.168043 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:49.168050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:49.168431 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:49.168908 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.168922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.171116 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.171122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.172769 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.172786 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.173866 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.173873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.174129 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.174898 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:49.174923 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:49.176931 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:49.179573 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:49.179579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:49.179591 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:49.179920 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:49.180331 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:49.180334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:49.180335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:49.180500 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:49.181219 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:49.181224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:49.181483 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 20:13:49.181680 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 20:13:49.182041 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":178,"threads_started":1}
I20260926 20:13:49.183856 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.164774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:60479 (local address 127.23.50.190:42405)
0926 20:13:49.165808 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:49.165828 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:49.165897 (+ 69us) server_negotiation.cc:407] Connection header received
0926 20:13:49.167056 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:49.167072 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:49.167568 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:49.167991 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:49.169092 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.170896 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.172932 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.173660 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.173890 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.177157 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:49.177164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:49.177181 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:49.177215 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:49.179433 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:49.180053 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:49.180058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:49.180063 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:49.180217 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:49.180605 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:49.180608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:49.180609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:49.181039 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:49.181164 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:49.182622 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0926 20:13:49.182980 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":108,"threads_started":1}
I20260926 20:13:49.184314 7351 ts_tablet_manager.cc:1397] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.004s
I20260926 20:13:49.186570 7459 heartbeater.cc:347] Connected to a master server at 127.23.50.190:42405
I20260926 20:13:49.186800 7459 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:49.187353 7459 heartbeater.cc:511] Master 127.23.50.190:42405 requested a full tablet report, sending...
I20260926 20:13:49.187114 7351 raft_consensus.cc:348] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:49.187595 7351 raft_consensus.cc:729] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01868c6c2ec4413d87294a208538e121, State: Initialized, Role: FOLLOWER
I20260926 20:13:49.188115 7351 consensus_queue.cc:260] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:49.189108 7351 ts_tablet_manager.cc:1428] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:13:49.190778 7270 ts_manager.cc:194] Registered new tserver with Master: 01868c6c2ec4413d87294a208538e121 (127.23.50.129:45491)
I20260926 20:13:49.194511 7270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:60479
I20260926 20:13:49.197199 7459 heartbeater.cc:503] Master 127.23.50.190:42405 was elected leader, sending a full tablet report...
Sep 26 20:13:49 dist-test-slave-7034 krb5kdc[6688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453628, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.129@KRBTEST.COM
I20260926 20:13:49.206967 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.193568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:45491 (local address 127.0.0.1:58054)
0926 20:13:49.194027 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:49.194074 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:49.194209 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:49.194847 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:49.194860 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:49.195312 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:49.195441 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.195450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.198637 (+ 3187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.198642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.199700 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.199712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.200349 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.200352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.200466 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.201075 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:49.201091 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:49.202809 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:49.205122 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:49.205129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:49.205133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:49.205452 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:49.205776 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:49.205778 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:49.205779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:49.205914 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:49.206434 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:49.206439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:49.206545 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 20:13:49.206661 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 20:13:49.206827 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":123,"threads_started":1}
I20260926 20:13:49.207135 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.193667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.23.50.129:45491)
0926 20:13:49.194385 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:49.194397 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:49.194420 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:13:49.194490 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:49.194495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:49.194689 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:49.194836 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:49.197402 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.198502 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.199858 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.200184 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.200323 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.202955 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:49.202964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:49.202975 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:49.202998 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:49.204932 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:49.205570 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:49.205576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:49.205578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:49.205676 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:49.206001 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:49.206004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:49.206006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:49.206320 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:49.206454 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:49.206795 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 20:13:49.206975 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":127,"threads_started":1}
I20260926 20:13:49.208405 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet e1bc3cf4d25f453897b49e09063421ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:28 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:58054
I20260926 20:13:49.209300 7468 tablet_replica.cc:330] stopping tablet replica
I20260926 20:13:49.209602 7468 raft_consensus.cc:2229] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:13:49.209916 7468 raft_consensus.cc:2258] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:49.210973 7468 ts_tablet_manager.cc:1905] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:49.214650 7468 ts_tablet_manager.cc:1918] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:13:49.214787 7468 log.cc:1196] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal/wals/e1bc3cf4d25f453897b49e09063421ba
I20260926 20:13:49.215107 7468 ts_tablet_manager.cc:1939] T e1bc3cf4d25f453897b49e09063421ba P 01868c6c2ec4413d87294a208538e121: Deleting consensus metadata
I20260926 20:13:49.217033 7257 catalog_manager.cc:4841] TS 01868c6c2ec4413d87294a208538e121 (127.23.50.129:45491): tablet e1bc3cf4d25f453897b49e09063421ba (table test-table [id=2477da5de1994a8991249859214d2dce]) successfully deleted
W20260926 20:13:49.289170 7463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:49.289435 7463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:49.289477 7463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:49.293978 7463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:49.294034 7463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:49.294131 7463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:13:49.299432 7463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:39265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=35335
--tserver_master_addrs=127.23.50.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:49.300452 7463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:49.301462 7463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:49.312424 7474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:49.312386 7473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:49.312386 7476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:49.312630 7463 server_base.cc:1034] running on GCE node
I20260926 20:13:49.313047 7463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:49.313725 7463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:49.314901 7463 hybrid_clock.cc:648] HybridClock initialized: now 1790453629314886 us; error 35 us; skew 500 ppm
Sep 26 20:13:49 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453629, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:49.317783 7463 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:13:49.318727 7463 webserver.cc:460] Webserver started at http://127.23.50.130:35335/ using document root <none> and password file <none>
I20260926 20:13:49.319300 7463 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:49.319398 7463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:49.323374 7463 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:13:49.325577 7483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:49.326860 7463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:13:49.327001 7463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "3ab5b58d10b4417c91f6271bc3a3d452"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "12db4c52cdbbb13bfeb642638465e1b7"
server_key_iv: "7168c70e3a84689ba27fc7586a47ae5b"
server_key_version: "encryptionkey@0"
I20260926 20:13:49.327551 7463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:49.361883 7463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:49.365243 7463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:49.365463 7463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:49.366119 7463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:49.367192 7490 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 20:13:49.371012 7490 ts_tablet_manager.cc:1852] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 20:13:49.371428 7490 ts_tablet_manager.cc:1905] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:49.375950 7490 ts_tablet_manager.cc:1918] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:13:49.376163 7490 ts_tablet_manager.cc:1939] T e1bc3cf4d25f453897b49e09063421ba P 3ab5b58d10b4417c91f6271bc3a3d452: Deleting consensus metadata
I20260926 20:13:49.376834 7463 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 20:13:49.376895 7463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260926 20:13:49.377079 7463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:49.377123 7463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:49.387089 7463 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:39265
I20260926 20:13:49.387120 7597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:39265 every 8 connection(s)
I20260926 20:13:49.388063 7463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:13:49.394168 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 7463
I20260926 20:13:49.394654 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:34853
--local_ip_for_outbound_sockets=127.23.50.131
--tserver_master_addrs=127.23.50.190:42405
--webserver_port=43241
--webserver_interface=127.23.50.131
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:13:49 dist-test-slave-7034 krb5kdc[6688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453629, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:49.403096 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.390074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:60589 (local address 127.23.50.190:42405)
0926 20:13:49.390287 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:49.390291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:49.391147 (+ 856us) server_negotiation.cc:407] Connection header received
0926 20:13:49.392120 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:49.392124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:49.392180 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:49.392288 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:49.393518 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.394162 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.395471 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.395710 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.395829 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.398611 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:49.398618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:49.398624 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:49.398662 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:49.400593 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:49.401153 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:49.401159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:49.401163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:49.401276 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:49.401648 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:49.401654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:49.401657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:49.401965 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:49.402083 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:49.402849 (+ 766us) server_negotiation.cc:299] Negotiation successful
0926 20:13:49.402970 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 20:13:49.404507 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.390484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:60589)
0926 20:13:49.390967 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:49.391012 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:49.391905 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:49.392471 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:49.392478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:49.392833 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:49.393360 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.393374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.394302 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.394305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.395324 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.395335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.395876 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.395880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.396055 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.396685 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:49.396838 (+ 153us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:49.398412 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:49.400742 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:49.400747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:49.400762 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:49.401042 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:49.401397 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:49.401400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:49.401401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:49.401544 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:49.402152 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:49.402163 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:49.402554 (+ 391us) client_negotiation.cc:769] Sending connection context
0926 20:13:49.402817 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 20:13:49.403238 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260926 20:13:49.406030 7598 heartbeater.cc:347] Connected to a master server at 127.23.50.190:42405
I20260926 20:13:49.406311 7598 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:49.406840 7598 heartbeater.cc:511] Master 127.23.50.190:42405 requested a full tablet report, sending...
I20260926 20:13:49.407996 7270 ts_manager.cc:194] Registered new tserver with Master: 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265)
I20260926 20:13:49.408829 7270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:60589
W20260926 20:13:49.509343 7601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:13:49.509603 7601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:13:49.509647 7601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:13:49.513794 7601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:13:49.513877 7601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:13:49.513937 7601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:13:49.519274 7601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:34853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=43241
--tserver_master_addrs=127.23.50.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:13:49.520309 7601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:13:49.521272 7601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:13:49.531129 7606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:49.531109 7607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:13:49.531100 7609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:13:49.531150 7601 server_base.cc:1034] running on GCE node
I20260926 20:13:49.531831 7601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:13:49.532492 7601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:13:49.533675 7601 hybrid_clock.cc:648] HybridClock initialized: now 1790453629533660 us; error 36 us; skew 500 ppm
Sep 26 20:13:49 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453629, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:13:49.536463 7601 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:13:49.537357 7601 webserver.cc:460] Webserver started at http://127.23.50.131:43241/ using document root <none> and password file <none>
I20260926 20:13:49.537935 7601 fs_manager.cc:359] Metadata directory not provided
I20260926 20:13:49.538013 7601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:13:49.541708 7601 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:13:49.543712 7616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:49.544970 7601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:13:49.545085 7601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "e2671a5362ec44c69479de3428f9977a"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "f0368eb60fbe90844729e7e92200442d"
server_key_iv: "5600945550f4bd1fbe35f0a2cf4057b6"
server_key_version: "encryptionkey@0"
I20260926 20:13:49.545621 7601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:13:49.573102 7601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:13:49.577153 7601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:13:49.577405 7601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:13:49.578121 7601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:13:49.579095 7601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:13:49.579145 7601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:49.579186 7601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:13:49.579228 7601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:13:49.588954 7601 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:34853
I20260926 20:13:49.589012 7729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:34853 every 8 connection(s)
I20260926 20:13:49.590123 7601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:13:49.591774 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 7601
Sep 26 20:13:49 dist-test-slave-7034 krb5kdc[6688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453629, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:13:49.602252 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.592469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.23.50.190:42405)
0926 20:13:49.592668 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:49.592671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:49.593006 (+ 335us) server_negotiation.cc:407] Connection header received
0926 20:13:49.593162 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:49.593165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:49.593221 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:49.593324 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:49.594425 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.595011 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.596261 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.596597 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.596680 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.598556 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:49.598559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:49.598561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:49.598585 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:49.600297 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:49.600795 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:49.600797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:49.600798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:49.600878 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:49.601210 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:49.601214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:49.601215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:49.601406 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:49.601640 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:49.601891 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 20:13:49.602046 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260926 20:13:49.606246 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.592157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:57637 (local address 127.23.50.190:42405)
0926 20:13:49.592320 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:49.592324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:49.593380 (+ 1056us) server_negotiation.cc:407] Connection header received
0926 20:13:49.595088 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:49.595092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:49.595148 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:49.595222 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:49.596771 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.597512 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.599119 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.599360 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.599479 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.602316 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:49.602330 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:49.602335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:49.602361 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:49.604076 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:49.604691 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:49.604695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:49.604697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:49.604785 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:49.605172 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:49.605174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:49.605176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:49.605369 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:49.605443 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:49.606033 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 20:13:49.606130 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 20:13:49.607146 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.592675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:57637)
0926 20:13:49.593226 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:49.593271 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:49.594874 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:49.595445 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:49.595455 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:49.595945 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:13:49.596610 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.596626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.597644 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.597649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.598982 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.598993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.599524 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.599529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.599719 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.600256 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:13:49.600273 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:13:49.602117 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:13:49.604196 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:49.604208 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:49.604222 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:49.604571 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:49.604875 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:13:49.604877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:13:49.604879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:13:49.605074 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:13:49.605459 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:13:49.605464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:13:49.605753 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 20:13:49.605976 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 20:13:49.606277 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":114,"threads_started":1}
I20260926 20:13:49.608453 7730 heartbeater.cc:347] Connected to a master server at 127.23.50.190:42405
I20260926 20:13:49.608736 7730 heartbeater.cc:464] Registering TS with master...
I20260926 20:13:49.609261 7730 heartbeater.cc:511] Master 127.23.50.190:42405 requested a full tablet report, sending...
I20260926 20:13:49.610347 7270 ts_manager.cc:194] Registered new tserver with Master: e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853)
I20260926 20:13:49.610970 7270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:57637
I20260926 20:13:49.616514 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:13:49.620522 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.617146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.23.50.190:42405)
0926 20:13:49.617306 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:49.617312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:49.617454 (+ 142us) server_negotiation.cc:407] Connection header received
0926 20:13:49.617591 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:49.617593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:49.617690 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:49.617777 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:13:49.618042 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.618553 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.619432 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.619635 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.619709 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.620056 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:13:49.620186 (+ 130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 20:13:49.620390 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40164: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 20:13:49.620597 7462 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40164: token expired
W20260926 20:13:49.620929 7196 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.50.190:42405)
I20260926 20:13:49.628433 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.621383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.23.50.190:42405)
0926 20:13:49.621496 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:49.621499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:49.621512 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:13:49.621599 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:49.621601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:49.621633 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:49.621718 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:13:49.622373 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.622840 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.623716 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.623919 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.623995 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.625372 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:13:49.625375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:13:49.625377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:13:49.625399 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:13:49.626915 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:49.627348 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:49.627350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:49.627351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:49.627423 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:13:49.627655 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:13:49.627657 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:13:49.627659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:13:49.627821 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:13:49.627921 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:49.628130 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 20:13:49.628220 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:13:49.631202 7270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:13:49.635051 7270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:13:49.643800 7393 tablet_service.cc:1467] Processing CreateTablet for tablet 30166f22c0b343c99c892e007ef535a8 (DEFAULT_TABLE table=test-table [id=c622a38aa3164013bc047058891e3008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:49.644517 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30166f22c0b343c99c892e007ef535a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:49.647529 7351 tablet_bootstrap.cc:492] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: Bootstrap starting.
I20260926 20:13:49.648573 7351 tablet_bootstrap.cc:654] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:49.648675 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.642616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:39265 (local address 127.0.0.1:46876)
0926 20:13:49.642814 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:49.642831 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:49.642926 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:49.643989 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:49.643992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:49.644013 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:49.644132 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.644139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.645641 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.645644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.646906 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.646917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.648138 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.648142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.648282 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.648336 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 20:13:49.648418 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:13:49.648509 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":1}
I20260926 20:13:49.649029 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.642709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:34853 (local address 127.0.0.1:47134)
0926 20:13:49.642970 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:49.642984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:49.643077 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:49.644522 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:49.644525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:49.644540 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:49.644677 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.644684 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.646132 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.646137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.646972 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.646981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.648420 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.648425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.648517 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.648536 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:13:49.648606 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:13:49.648662 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":125,"threads_started":1}
I20260926 20:13:49.649036 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.643027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.23.50.130:39265)
0926 20:13:49.643457 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:49.643469 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:49.643494 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:13:49.643577 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:49.643583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:49.643817 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:49.644048 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:49.644242 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.645505 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.647139 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.647985 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.648126 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.648299 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:49.648761 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 20:13:49.648885 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":162,"threads_started":1}
I20260926 20:13:49.649525 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.643475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.23.50.131:34853)
0926 20:13:49.643782 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:49.643798 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:49.643832 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:13:49.643897 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:49.643902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:49.644148 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:49.644347 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:49.644792 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.646002 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.647110 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.648110 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.648228 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.648375 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:49.649028 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 20:13:49.649372 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":121,"threads_started":1}
I20260926 20:13:49.650763 7351 tablet_bootstrap.cc:492] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: No bootstrap required, opened a new log
I20260926 20:13:49.650884 7351 ts_tablet_manager.cc:1397] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 20:13:49.651350 7351 raft_consensus.cc:348] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:49.651484 7351 raft_consensus.cc:374] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:49.651037 7532 tablet_service.cc:1467] Processing CreateTablet for tablet 30166f22c0b343c99c892e007ef535a8 (DEFAULT_TABLE table=test-table [id=c622a38aa3164013bc047058891e3008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:49.651537 7351 raft_consensus.cc:729] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01868c6c2ec4413d87294a208538e121, State: Initialized, Role: FOLLOWER
I20260926 20:13:49.651674 7351 consensus_queue.cc:260] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:49.651968 7351 ts_tablet_manager.cc:1428] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 20:13:49.652115 7532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30166f22c0b343c99c892e007ef535a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:49.652634 7664 tablet_service.cc:1467] Processing CreateTablet for tablet 30166f22c0b343c99c892e007ef535a8 (DEFAULT_TABLE table=test-table [id=c622a38aa3164013bc047058891e3008]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:13:49.653645 7664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30166f22c0b343c99c892e007ef535a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:13:49.656726 7490 tablet_bootstrap.cc:492] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452: Bootstrap starting.
I20260926 20:13:49.659049 7490 tablet_bootstrap.cc:654] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:49.659786 7490 log.cc:824] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:49.661154 7744 tablet_bootstrap.cc:492] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a: Bootstrap starting.
I20260926 20:13:49.661835 7490 tablet_bootstrap.cc:492] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452: No bootstrap required, opened a new log
I20260926 20:13:49.662060 7490 ts_tablet_manager.cc:1397] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
W20260926 20:13:49.663383 7460 tablet.cc:2369] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:13:49.663775 7744 tablet_bootstrap.cc:654] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a: Neither blocks nor log segments found. Creating new log.
I20260926 20:13:49.665086 7744 log.cc:824] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a: Log is configured to *not* fsync() on all Append() calls
I20260926 20:13:49.666149 7490 raft_consensus.cc:348] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:49.666409 7490 raft_consensus.cc:374] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:49.666471 7490 raft_consensus.cc:729] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab5b58d10b4417c91f6271bc3a3d452, State: Initialized, Role: FOLLOWER
I20260926 20:13:49.666952 7490 consensus_queue.cc:260] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:49.667204 7744 tablet_bootstrap.cc:492] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a: No bootstrap required, opened a new log
I20260926 20:13:49.667404 7744 ts_tablet_manager.cc:1397] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:13:49.668015 7490 ts_tablet_manager.cc:1428] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 20:13:49.668404 7598 heartbeater.cc:503] Master 127.23.50.190:42405 was elected leader, sending a full tablet report...
I20260926 20:13:49.671029 7744 raft_consensus.cc:348] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:49.671327 7744 raft_consensus.cc:374] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:13:49.671391 7744 raft_consensus.cc:729] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2671a5362ec44c69479de3428f9977a, State: Initialized, Role: FOLLOWER
I20260926 20:13:49.671850 7744 consensus_queue.cc:260] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:49.672564 7730 heartbeater.cc:503] Master 127.23.50.190:42405 was elected leader, sending a full tablet report...
I20260926 20:13:49.672640 7744 ts_tablet_manager.cc:1428] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 20:13:49.841122 7731 tablet.cc:2369] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:13:49.889367 7599 tablet.cc:2369] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:13:49.903842 7747 raft_consensus.cc:482] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:13:49.904067 7747 raft_consensus.cc:504] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:49.905429 7747 leader_election.cc:288] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265), 01868c6c2ec4413d87294a208538e121 (127.23.50.129:45491)
I20260926 20:13:49.909446 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.905999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:38223 (local address 127.23.50.130:39265)
0926 20:13:49.906177 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:49.906182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:49.906340 (+ 158us) server_negotiation.cc:407] Connection header received
0926 20:13:49.906491 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:49.906493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:49.906536 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:49.906619 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:49.906887 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.907370 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.908464 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.908855 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.908994 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.909051 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:49.909247 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 20:13:49.909328 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:13:49.909488 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.905720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:45491 (local address 127.23.50.131:49413)
0926 20:13:49.905870 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:49.905887 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:49.905992 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:49.906453 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:49.906456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:49.906473 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:49.906600 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.906605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.907372 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.907378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.908320 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.908331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.909037 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.909041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.909168 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.909187 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:13:49.909274 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 20:13:49.909351 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":78}
I20260926 20:13:49.909489 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.905901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:49413 (local address 127.23.50.129:45491)
0926 20:13:49.906174 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:49.906178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:49.906192 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:13:49.906241 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:49.906243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:49.906296 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:49.906430 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:13:49.906738 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.907257 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.908463 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.908908 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.909049 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.909219 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:49.909299 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 20:13:49.909372 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260926 20:13:49.909881 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.905720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:39265 (local address 127.23.50.131:38223)
0926 20:13:49.906248 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:49.906259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:13:49.906372 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:13:49.906637 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:13:49.906640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:13:49.906655 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:13:49.906777 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.906782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.907478 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.907484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.908334 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:13:49.908343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.908989 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.908992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:13:49.909098 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.909123 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:13:49.909627 (+ 504us) client_negotiation.cc:240] Negotiation successful
0926 20:13:49.909725 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":56,"thread_start_us":144,"threads_started":1}
I20260926 20:13:49.910254 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30166f22c0b343c99c892e007ef535a8" candidate_uuid: "e2671a5362ec44c69479de3428f9977a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01868c6c2ec4413d87294a208538e121" is_pre_election: true
I20260926 20:13:49.910614 7413 raft_consensus.cc:2454] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2671a5362ec44c69479de3428f9977a in term 0.
I20260926 20:13:49.910609 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30166f22c0b343c99c892e007ef535a8" candidate_uuid: "e2671a5362ec44c69479de3428f9977a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" is_pre_election: true
I20260926 20:13:49.910879 7552 raft_consensus.cc:2454] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2671a5362ec44c69479de3428f9977a in term 0.
I20260926 20:13:49.911139 7620 leader_election.cc:302] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01868c6c2ec4413d87294a208538e121, e2671a5362ec44c69479de3428f9977a; no voters:
I20260926 20:13:49.911437 7747 raft_consensus.cc:2781] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:13:49.911521 7747 raft_consensus.cc:482] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:13:49.911553 7747 raft_consensus.cc:3037] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:49.912657 7747 raft_consensus.cc:504] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:49.913026 7747 leader_election.cc:288] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [CANDIDATE]: Term 1 election: Requested vote from peers 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265), 01868c6c2ec4413d87294a208538e121 (127.23.50.129:45491)
I20260926 20:13:49.913412 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30166f22c0b343c99c892e007ef535a8" candidate_uuid: "e2671a5362ec44c69479de3428f9977a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01868c6c2ec4413d87294a208538e121"
I20260926 20:13:49.913494 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30166f22c0b343c99c892e007ef535a8" candidate_uuid: "e2671a5362ec44c69479de3428f9977a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ab5b58d10b4417c91f6271bc3a3d452"
I20260926 20:13:49.913554 7413 raft_consensus.cc:3037] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:49.913672 7552 raft_consensus.cc:3037] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:13:49.914688 7413 raft_consensus.cc:2454] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2671a5362ec44c69479de3428f9977a in term 1.
I20260926 20:13:49.914784 7552 raft_consensus.cc:2454] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2671a5362ec44c69479de3428f9977a in term 1.
I20260926 20:13:49.915094 7620 leader_election.cc:302] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01868c6c2ec4413d87294a208538e121, e2671a5362ec44c69479de3428f9977a; no voters:
I20260926 20:13:49.915297 7747 raft_consensus.cc:2781] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:13:49.915541 7747 raft_consensus.cc:686] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 1 LEADER]: Becoming Leader. State: Replica: e2671a5362ec44c69479de3428f9977a, State: Running, Role: LEADER
I20260926 20:13:49.915941 7747 consensus_queue.cc:237] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:13:49.919147 7268 catalog_manager.cc:5495] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a reported cstate change: term changed from 0 to 1, leader changed from <none> to e2671a5362ec44c69479de3428f9977a (127.23.50.131). New cstate: current_term: 1 leader_uuid: "e2671a5362ec44c69479de3428f9977a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } health_report { overall_health: UNKNOWN } } }
I20260926 20:13:49.945266 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.941728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47138 (local address 127.23.50.131:34853)
0926 20:13:49.941900 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 20:13:49.941905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:13:49.941921 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:13:49.942056 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:13:49.942058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:13:49.942115 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:13:49.942230 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:13:49.942466 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.942975 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.943807 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:13:49.944024 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:13:49.944136 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:13:49.944452 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:13:49.944857 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:13:49.944986 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 20:13:49.945060 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 20:13:49.945118 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:13:49.951614 7552 raft_consensus.cc:1261] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452 [term 1 FOLLOWER]: Refusing update from remote peer e2671a5362ec44c69479de3428f9977a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:13:49.951614 7413 raft_consensus.cc:1261] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Refusing update from remote peer e2671a5362ec44c69479de3428f9977a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:13:49.952312 7747 consensus_queue.cc:1035] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:13:49.952579 7750 consensus_queue.cc:1035] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [LEADER]: Connected to new peer: Peer: permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:13:49.958873 7754 mvcc.cc:205] Tried to move back new op lower bound from 7333698068273209344 to 7333698068138360832. Current Snapshot: MvccSnapshot[applied={T|T < 7333698068273209344}]
I20260926 20:13:49.962379 7753 mvcc.cc:205] Tried to move back new op lower bound from 7333698068273209344 to 7333698068138360832. Current Snapshot: MvccSnapshot[applied={T|T < 7333698068273209344 or (T in {7333698068273209344})}]
I20260926 20:13:49.968482 7270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:13:49.968684 7270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:13:49.971638 7270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Sending DeleteTablet for 3 replicas of tablet 30166f22c0b343c99c892e007ef535a8
I20260926 20:13:49.972561 7532 tablet_service.cc:1514] Processing DeleteTablet for tablet 30166f22c0b343c99c892e007ef535a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:49 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:46876
I20260926 20:13:49.972628 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet 30166f22c0b343c99c892e007ef535a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:49 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:58054
I20260926 20:13:49.972611 7664 tablet_service.cc:1514] Processing DeleteTablet for tablet 30166f22c0b343c99c892e007ef535a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:13:49 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:47134
I20260926 20:13:49.973258 7763 tablet_replica.cc:330] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: stopping tablet replica
I20260926 20:13:49.973412 7763 raft_consensus.cc:2229] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:13:49.973415 7764 tablet_replica.cc:330] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a: stopping tablet replica
I20260926 20:13:49.973454 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 7308
I20260926 20:13:49.973587 7763 raft_consensus.cc:2258] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:49.973917 7764 raft_consensus.cc:2229] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:13:49.974431 7764 raft_consensus.cc:2258] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:49.974889 7765 tablet_replica.cc:330] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452: stopping tablet replica
I20260926 20:13:49.975273 7765 raft_consensus.cc:2229] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:13:49.975651 7765 raft_consensus.cc:2258] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:13:49.976229 7764 ts_tablet_manager.cc:1905] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:49.977123 7765 ts_tablet_manager.cc:1905] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:13:49.979221 7764 ts_tablet_manager.cc:1918] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:13:49.980020 7765 ts_tablet_manager.cc:1918] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:13:49.980118 7765 log.cc:1196] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/30166f22c0b343c99c892e007ef535a8
I20260926 20:13:49.980458 7765 ts_tablet_manager.cc:1939] T 30166f22c0b343c99c892e007ef535a8 P 3ab5b58d10b4417c91f6271bc3a3d452: Deleting consensus metadata
W20260926 20:13:49.980984 7257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:45491 (error 108)
I20260926 20:13:49.981627 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 7463
I20260926 20:13:49.981876 7764 log.cc:1196] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/30166f22c0b343c99c892e007ef535a8
I20260926 20:13:49.982576 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.981818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:45491 (local address 127.0.0.1:58056)
0926 20:13:49.982388 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:49.982470 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:45491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466}
I20260926 20:13:49.983150 7764 ts_tablet_manager.cc:1939] T 30166f22c0b343c99c892e007ef535a8 P e2671a5362ec44c69479de3428f9977a: Deleting consensus metadata
W20260926 20:13:49.984352 7257 catalog_manager.cc:4584] TS 01868c6c2ec4413d87294a208538e121 (127.23.50.129:45491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30166f22c0b343c99c892e007ef535a8: Network error: Client connection negotiation failed: client connection to 127.23.50.129:45491: connect: Connection refused (error 111)
I20260926 20:13:49.984601 7257 catalog_manager.cc:4841] TS e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853): tablet 30166f22c0b343c99c892e007ef535a8 (table test-table [id=c622a38aa3164013bc047058891e3008]) successfully deleted
I20260926 20:13:49.989679 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 7601
I20260926 20:13:49.989887 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:13:49.989550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:39265 (local address 127.0.0.1:46892)
0926 20:13:49.989731 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:13:49.989784 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.130:39265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 20:13:49.995982 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 7237
I20260926 20:14:10.002974 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42405
--webserver_interface=127.23.50.190
--webserver_port=39949
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:42405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:10.117470 7767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:10.117746 7767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:10.117789 7767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:10.121927 7767 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 20:14:10.121989 7767 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 20:14:10.122004 7767 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 20:14:10.122018 7767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:10.122062 7767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:10.122097 7767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:10.122136 7767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:10.122170 7767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:10.127507 7767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:42405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=39949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:10.128670 7767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:10.129731 7767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:10.138633 7773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:10.138653 7772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:10.138672 7775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:10.138965 7767 server_base.cc:1034] running on GCE node
I20260926 20:14:10.139698 7767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:10.140851 7767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:10.142017 7767 hybrid_clock.cc:648] HybridClock initialized: now 1790453650141979 us; error 51 us; skew 500 ppm
Sep 26 20:14:10 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453650, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:10.148861 7767 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:14:10.149920 7767 webserver.cc:460] Webserver started at http://127.23.50.190:39949/ using document root <none> and password file <none>
I20260926 20:14:10.150547 7767 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:10.150648 7767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:10.154862 7767 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:14:10.157315 7782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:10.158725 7767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:14:10.158881 7767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "c6d41a7493584b43a737d96c60ea6ca9"
format_stamp: "Formatted at 2026-09-26 20:13:27 on dist-test-slave-7034"
server_key: "376b3f9b8983fc140d73c43fb6dd738d"
server_key_iv: "8b7818ec3ff565a9dca1d44c4ecdca49"
server_key_version: "encryptionkey@0"
I20260926 20:14:10.159472 7767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:10.191200 7767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:10.194988 7767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:10.195209 7767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:10.202683 7767 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:42405
I20260926 20:14:10.202971 7834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:42405 every 8 connection(s)
I20260926 20:14:10.203938 7767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:10.208514 7835 sys_catalog.cc:263] Verifying existing consensus state
I20260926 20:14:10.210362 7835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Bootstrap starting.
I20260926 20:14:10.210446 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 7767
I20260926 20:14:10.211072 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:45491
--local_ip_for_outbound_sockets=127.23.50.129
--tserver_master_addrs=127.23.50.190:42405
--webserver_port=32771
--webserver_interface=127.23.50.129
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:14:10.229127 7835 log.cc:824] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:10.240747 7835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:14:10.241267 7835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Bootstrap complete.
I20260926 20:14:10.245514 7835 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } }
I20260926 20:14:10.245970 7835 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d41a7493584b43a737d96c60ea6ca9, State: Initialized, Role: FOLLOWER
I20260926 20:14:10.246472 7835 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } }
I20260926 20:14:10.246587 7835 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:10.246677 7835 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:10.246802 7835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 2 FOLLOWER]: Advancing to term 3
I20260926 20:14:10.248911 7835 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } }
I20260926 20:14:10.249269 7835 leader_election.cc:302] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6d41a7493584b43a737d96c60ea6ca9; no voters:
I20260926 20:14:10.249607 7835 leader_election.cc:288] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 20:14:10.249703 7839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 20:14:10.249918 7839 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [term 3 LEADER]: Becoming Leader. State: Replica: c6d41a7493584b43a737d96c60ea6ca9, State: Running, Role: LEADER
I20260926 20:14:10.250278 7839 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } }
I20260926 20:14:10.250892 7835 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:10.251236 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6d41a7493584b43a737d96c60ea6ca9. Latest consensus state: current_term: 3 leader_uuid: "c6d41a7493584b43a737d96c60ea6ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } } }
I20260926 20:14:10.251439 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:10.251614 7840 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c6d41a7493584b43a737d96c60ea6ca9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d41a7493584b43a737d96c60ea6ca9" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 42405 } } }
I20260926 20:14:10.251734 7840 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:10.251924 7843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:10.257310 7843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:10.258190 7843 catalog_manager.cc:1259] Loaded cluster ID: 8a2581fa9bc1487689332390b9d01a52
I20260926 20:14:10.258260 7843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:10.260782 7843 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:10.261566 7843 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Loaded TSK: 0
I20260926 20:14:10.261662 7843 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6d41a7493584b43a737d96c60ea6ca9: Loaded TSK: 1
F20260926 20:14:10.264331 7843 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f833cd219ad google::LogMessage::Fail() at ??:0
@ 0x7f833cd2488a google::LogMessage::SendToLog() at ??:0
@ 0x7f833cd214de google::LogMessage::Flush() at ??:0
@ 0x7f833cd22a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f833f9e518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f833f9ddf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f834768a7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f834766499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8347665178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f83476a6df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f83476c582b std::function<>::operator()() at ??:0
@ 0x7f8347664b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8347665f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8347662579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f83476a6640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f833ef733d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f833ef73c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f833ef753f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f833ef64512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f83405a86db start_thread at ??:0
@ 0x7f833b81f61f clone at ??:0
W20260926 20:14:10.352156 7837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:10.352512 7837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:10.352584 7837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:10.359484 7837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:10.359577 7837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:10.359697 7837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:14:10.368397 7837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:45491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=32771
--tserver_master_addrs=127.23.50.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:10.369971 7837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:10.371318 7837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:10.383898 7882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:10.384194 7883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:10.383915 7885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:10.385586 7837 server_base.cc:1034] running on GCE node
I20260926 20:14:10.386118 7837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:10.386960 7837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:10.388170 7837 hybrid_clock.cc:648] HybridClock initialized: now 1790453650388146 us; error 34 us; skew 500 ppm
Sep 26 20:14:10 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453650, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:10.391636 7837 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:14:10.392766 7837 webserver.cc:460] Webserver started at http://127.23.50.129:32771/ using document root <none> and password file <none>
I20260926 20:14:10.393482 7837 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:10.393575 7837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:10.398854 7837 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:14:10.401234 7892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:10.402704 7837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:14:10.402844 7837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "01868c6c2ec4413d87294a208538e121"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "84dac65d01fa17d8bd319aa547640d34"
server_key_iv: "fd0b63ab5cffb5dc6d1aab0875cfc6a6"
server_key_version: "encryptionkey@0"
I20260926 20:14:10.403419 7837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:10.418344 7837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:10.421648 7837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:10.421922 7837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:10.422525 7837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:10.423686 7899 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 20:14:10.429433 7837 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 20:14:10.429553 7837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 20:14:10.429673 7837 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 20:14:10.431449 7837 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 20:14:10.431524 7837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 20:14:10.431874 7899 tablet_bootstrap.cc:492] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: Bootstrap starting.
I20260926 20:14:10.442312 7837 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:45491
I20260926 20:14:10.442574 8006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:45491 every 8 connection(s)
I20260926 20:14:10.443429 7837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:14:10.448623 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 7837
I20260926 20:14:10.449163 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:39265
--local_ip_for_outbound_sockets=127.23.50.130
--tserver_master_addrs=127.23.50.190:42405
--webserver_port=35335
--webserver_interface=127.23.50.130
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:14:10.454654 7899 log.cc:824] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:10.462131 7899 tablet_bootstrap.cc:492] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 20:14:10.462641 7899 tablet_bootstrap.cc:492] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: Bootstrap complete.
I20260926 20:14:10.463907 7899 ts_tablet_manager.cc:1397] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.002s
I20260926 20:14:10.467121 7899 raft_consensus.cc:348] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:14:10.467625 7899 raft_consensus.cc:729] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01868c6c2ec4413d87294a208538e121, State: Initialized, Role: FOLLOWER
I20260926 20:14:10.468137 7899 consensus_queue.cc:260] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:14:10.469226 7899 ts_tablet_manager.cc:1428] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260926 20:14:10.545123 8009 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:14:10.445440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:54349)
0926 20:14:10.445946 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:10.446000 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:10.446832 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:10.544280 (+ 97448us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":149,"threads_started":1}
W20260926 20:14:10.545794 8007 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:42405 (0 consecutive failures): Network error: Failed to ping master at 127.23.50.190:42405: Client connection negotiation failed: client connection to 127.23.50.190:42405: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 20:14:10.546614 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:10.546344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:60601)
0926 20:14:10.546475 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:10.546556 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 20:14:10.547330 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:10.547121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:58347)
0926 20:14:10.547225 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:10.547272 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 20:14:10.547531 8007 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:14:10.598881 8010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:10.599159 8010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:10.599237 8010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:10.603305 8010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:10.603365 8010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:10.603466 8010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:14:10.608994 8010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:39265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=35335
--tserver_master_addrs=127.23.50.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:10.610101 8010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:10.611155 8010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:10.620692 8020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:10.620779 8018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:10.620699 8017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:10.621002 8010 server_base.cc:1034] running on GCE node
I20260926 20:14:10.621410 8010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:10.622095 8010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:10.623265 8010 hybrid_clock.cc:648] HybridClock initialized: now 1790453650623246 us; error 29 us; skew 500 ppm
Sep 26 20:14:10 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453650, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:10.626145 8010 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:14:10.627017 8010 webserver.cc:460] Webserver started at http://127.23.50.130:35335/ using document root <none> and password file <none>
I20260926 20:14:10.627579 8010 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:10.627674 8010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:10.631789 8010 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:14:10.634042 8027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:10.635381 8010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:14:10.635522 8010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "3ab5b58d10b4417c91f6271bc3a3d452"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "12db4c52cdbbb13bfeb642638465e1b7"
server_key_iv: "7168c70e3a84689ba27fc7586a47ae5b"
server_key_version: "encryptionkey@0"
I20260926 20:14:10.636101 8010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:10.671408 8010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:10.674865 8010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:10.675076 8010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:10.675683 8010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:10.676539 8010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:10.676610 8010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:10.676685 8010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:10.676740 8010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:10.686100 8010 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:39265
I20260926 20:14:10.686156 8140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:39265 every 8 connection(s)
I20260926 20:14:10.687227 8010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:14:10.691373 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:10.689455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:48273)
0926 20:14:10.689987 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:10.690105 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260926 20:14:10.691526 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 8010
W20260926 20:14:10.691905 8141 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:42405 (0 consecutive failures): Network error: Failed to ping master at 127.23.50.190:42405: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
I20260926 20:14:10.692135 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:34853
--local_ip_for_outbound_sockets=127.23.50.131
--tserver_master_addrs=127.23.50.190:42405
--webserver_port=43241
--webserver_interface=127.23.50.131
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:14:10.692688 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:10.692388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:43853)
0926 20:14:10.692531 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:10.692599 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 20:14:10.693425 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:10.693140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:58011)
0926 20:14:10.693277 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:10.693345 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 20:14:10.693678 8141 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 20:14:10.804473 8144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:10.804808 8144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:10.804888 8144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:10.809182 8144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:10.809270 8144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:10.809362 8144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:14:10.814745 8144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:34853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=43241
--tserver_master_addrs=127.23.50.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:10.815863 8144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:10.816948 8144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:10.826690 8149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:10.826719 8150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:10.826720 8152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:10.826880 8144 server_base.cc:1034] running on GCE node
I20260926 20:14:10.827397 8144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:10.828058 8144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:10.829267 8144 hybrid_clock.cc:648] HybridClock initialized: now 1790453650829238 us; error 40 us; skew 500 ppm
Sep 26 20:14:10 dist-test-slave-7034 krb5kdc[6688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453650, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:10.832032 8144 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:14:10.832901 8144 webserver.cc:460] Webserver started at http://127.23.50.131:43241/ using document root <none> and password file <none>
I20260926 20:14:10.833415 8144 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:10.833474 8144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:10.837335 8144 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:14:10.839427 8159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:10.840765 8144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:14:10.840862 8144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "e2671a5362ec44c69479de3428f9977a"
format_stamp: "Formatted at 2026-09-26 20:13:28 on dist-test-slave-7034"
server_key: "f0368eb60fbe90844729e7e92200442d"
server_key_iv: "5600945550f4bd1fbe35f0a2cf4057b6"
server_key_version: "encryptionkey@0"
I20260926 20:14:10.841370 8144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:10.864279 8144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:10.867532 8144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:10.867697 8144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:10.868273 8144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:10.869190 8144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:10.869240 8144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:10.869290 8144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:10.869306 8144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:10.879699 8144 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:34853
I20260926 20:14:10.879720 8272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:34853 every 8 connection(s)
I20260926 20:14:10.880824 8144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:14:10.884996 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:10.883135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:37301)
0926 20:14:10.883658 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:10.883776 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":117,"threads_started":1}
W20260926 20:14:10.885588 8273 heartbeater.cc:644] Failed to heartbeat to 127.23.50.190:42405 (0 consecutive failures): Network error: Failed to ping master at 127.23.50.190:42405: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
I20260926 20:14:10.886286 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:10.886044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:44637)
0926 20:14:10.886165 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:10.886225 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 20:14:10.886843 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:10.886658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:37171)
0926 20:14:10.886754 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:10.886796 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 20:14:10.887008 8273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 20:14:10.889596 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 8144
I20260926 20:14:11.548736 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:11.548264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:56299)
0926 20:14:11.548539 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:11.548603 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":74,"threads_started":1}
I20260926 20:14:11.694694 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:11.694270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:48477)
0926 20:14:11.694536 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:11.694593 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260926 20:14:11.754675 8279 raft_consensus.cc:482] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:14:11.754920 8279 raft_consensus.cc:504] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:14:11.756138 8279 leader_election.cc:288] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265), e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853)
Sep 26 20:14:11 dist-test-slave-7034 krb5kdc[6688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453650, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.131@KRBTEST.COM
Sep 26 20:14:11 dist-test-slave-7034 krb5kdc[6688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453650, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.130@KRBTEST.COM
I20260926 20:14:11.770774 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:11.756452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:34853 (local address 127.23.50.129:52627)
0926 20:14:11.756989 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:11.757004 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:11.757105 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:11.759365 (+ 2260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:11.759372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:11.759533 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:11.759922 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:11.759938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:11.761232 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:11.761235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:11.762282 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:11.762293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:11.763001 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:11.763005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:11.763231 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:11.763905 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:14:11.763922 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:11.765600 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:11.767894 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:11.767900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:11.767914 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:11.768193 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:11.768616 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:11.768620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:11.768625 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:11.768810 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:11.769653 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:11.769660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:11.769992 (+ 332us) client_negotiation.cc:769] Sending connection context
0926 20:14:11.770151 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 20:14:11.770571 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":236,"spinlock_wait_cycles":30464,"thread_start_us":61,"threads_started":1}
I20260926 20:14:11.770790 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:11.756753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:55323 (local address 127.23.50.130:39265)
0926 20:14:11.757047 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:11.757085 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:11.757121 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:14:11.758041 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:11.758050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:11.758525 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:11.758871 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:11.760103 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:11.761066 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:11.762383 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:11.762869 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:11.763057 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:11.766263 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:11.766271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:11.766288 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:11.766324 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:11.768189 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:11.768710 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:11.768716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:11.768721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:11.768855 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:11.769152 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:11.769155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:11.769156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:11.769683 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:11.769872 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:11.770338 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 20:14:11.770613 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260926 20:14:11.770862 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:11.756579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:52627 (local address 127.23.50.131:34853)
0926 20:14:11.756879 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:11.756896 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:11.757120 (+ 224us) server_negotiation.cc:407] Connection header received
0926 20:14:11.758041 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:11.758050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:11.758525 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:11.758872 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:11.760102 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:11.761066 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:11.762407 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:11.762874 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:11.763056 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:11.765884 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:11.765892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:11.765913 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:11.765941 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:11.767759 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:11.768323 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:11.768336 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:11.768341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:11.768506 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:11.768931 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:11.768937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:11.768939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:11.769429 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:11.769535 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:11.770357 (+ 822us) server_negotiation.cc:299] Negotiation successful
0926 20:14:11.770670 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":102,"threads_started":1}
I20260926 20:14:11.770870 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:11.756347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:39265 (local address 127.23.50.129:55323)
0926 20:14:11.756584 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:11.756599 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:11.756709 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:11.758927 (+ 2218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:11.758933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:11.759406 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:11.759922 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:11.759938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:11.761222 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:11.761227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:11.762255 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:11.762266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:11.763001 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:11.763005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:11.763235 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:11.763943 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:14:11.763956 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:11.766027 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:11.768322 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:11.768325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:11.768327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:11.768548 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:11.768961 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:11.768963 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:11.768965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:11.769043 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:11.769834 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:11.769836 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:11.770011 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 20:14:11.770190 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 20:14:11.770677 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":257}
W20260926 20:14:11.773587 7894 leader_election.cc:341] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265): Not found: Tablet not found: 30166f22c0b343c99c892e007ef535a8
W20260926 20:14:11.773761 7896 leader_election.cc:341] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853): Not found: Tablet not found: 30166f22c0b343c99c892e007ef535a8
I20260926 20:14:11.773890 7896 leader_election.cc:302] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01868c6c2ec4413d87294a208538e121; no voters: 3ab5b58d10b4417c91f6271bc3a3d452, e2671a5362ec44c69479de3428f9977a
I20260926 20:14:11.774201 8279 raft_consensus.cc:2726] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:14:11.887984 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:11.887575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:51811)
0926 20:14:11.887825 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:11.887885 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":69,"threads_started":1}
I20260926 20:14:12.550100 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:12.549595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:41987)
0926 20:14:12.549929 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:12.549992 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20260926 20:14:12.695914 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:12.695502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:34695)
0926 20:14:12.695748 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:12.695821 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":76,"threads_started":1}
I20260926 20:14:12.889281 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:12.888847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:54115)
0926 20:14:12.889119 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:12.889172 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
I20260926 20:14:13.551378 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:13.550936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:33515)
0926 20:14:13.551200 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:13.551281 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260926 20:14:13.558729 8288 raft_consensus.cc:482] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:14:13.558874 8288 raft_consensus.cc:504] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:14:13.559372 8288 leader_election.cc:288] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265), e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853)
W20260926 20:14:13.560212 7896 leader_election.cc:341] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853): Not found: Tablet not found: 30166f22c0b343c99c892e007ef535a8
W20260926 20:14:13.560312 7894 leader_election.cc:341] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265): Not found: Tablet not found: 30166f22c0b343c99c892e007ef535a8
I20260926 20:14:13.560484 7894 leader_election.cc:302] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01868c6c2ec4413d87294a208538e121; no voters: 3ab5b58d10b4417c91f6271bc3a3d452, e2671a5362ec44c69479de3428f9977a
I20260926 20:14:13.560657 8288 raft_consensus.cc:2726] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:14:13.697358 8289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:13.696920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:39989)
0926 20:14:13.697193 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:13.697261 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260926 20:14:13.890672 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:13.890227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:39449)
0926 20:14:13.890519 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:13.890576 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260926 20:14:14.552695 8291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:14.552238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:44969)
0926 20:14:14.552528 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:14.552587 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
I20260926 20:14:14.698652 8292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:14.698206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:54877)
0926 20:14:14.698493 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:14.698557 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":73,"threads_started":1}
I20260926 20:14:14.892019 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:14.891566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:39287)
0926 20:14:14.891857 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:14.891919 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":77,"threads_started":1}
I20260926 20:14:15.554211 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:15.553677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:39651)
0926 20:14:15.554047 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:15.554111 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":76,"threads_started":1}
I20260926 20:14:15.699939 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:15.699550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:40753)
0926 20:14:15.699795 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:15.699848 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260926 20:14:15.893392 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:15.892942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:53801)
0926 20:14:15.893227 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:15.893295 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":84,"threads_started":1}
I20260926 20:14:16.218079 8297 raft_consensus.cc:482] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:14:16.218214 8297 raft_consensus.cc:504] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:14:16.218715 8297 leader_election.cc:288] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265), e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853)
W20260926 20:14:16.219535 7896 leader_election.cc:341] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853): Not found: Tablet not found: 30166f22c0b343c99c892e007ef535a8
W20260926 20:14:16.219627 7894 leader_election.cc:341] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265): Not found: Tablet not found: 30166f22c0b343c99c892e007ef535a8
I20260926 20:14:16.219746 7894 leader_election.cc:302] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01868c6c2ec4413d87294a208538e121; no voters: 3ab5b58d10b4417c91f6271bc3a3d452, e2671a5362ec44c69479de3428f9977a
I20260926 20:14:16.219942 8297 raft_consensus.cc:2726] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:14:16.555613 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:16.555091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:56335)
0926 20:14:16.555409 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:16.555498 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260926 20:14:16.701049 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:16.700665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:48459)
0926 20:14:16.700905 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:16.700959 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260926 20:14:16.894703 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:16.894275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:39519)
0926 20:14:16.894543 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:16.894607 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":71,"threads_started":1}
I20260926 20:14:17.556957 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:17.556497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:54079)
0926 20:14:17.556777 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:17.556849 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":80,"threads_started":1}
I20260926 20:14:17.702409 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:17.701944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:32911)
0926 20:14:17.702227 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:17.702315 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":69,"threads_started":1}
I20260926 20:14:17.895939 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:17.895485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:33175)
0926 20:14:17.895787 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:17.895846 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":72,"threads_started":1}
I20260926 20:14:18.558321 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:18.557819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:37641)
0926 20:14:18.558156 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:18.558216 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":80,"threads_started":1}
I20260926 20:14:18.703612 8305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:18.703216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:44001)
0926 20:14:18.703462 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:18.703516 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
I20260926 20:14:18.897300 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:18.896832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:53915)
0926 20:14:18.897109 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:18.897173 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":89,"threads_started":1}
I20260926 20:14:19.388444 8307 raft_consensus.cc:482] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:14:19.388576 8307 raft_consensus.cc:504] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:14:19.389035 8307 leader_election.cc:288] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265), e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853)
W20260926 20:14:19.389930 7894 leader_election.cc:341] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265): Not found: Tablet not found: 30166f22c0b343c99c892e007ef535a8
W20260926 20:14:19.390064 7896 leader_election.cc:341] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853): Not found: Tablet not found: 30166f22c0b343c99c892e007ef535a8
I20260926 20:14:19.390196 7896 leader_election.cc:302] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01868c6c2ec4413d87294a208538e121; no voters: 3ab5b58d10b4417c91f6271bc3a3d452, e2671a5362ec44c69479de3428f9977a
I20260926 20:14:19.390347 8307 raft_consensus.cc:2726] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:14:19.559665 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:19.559178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:41607)
0926 20:14:19.559493 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:19.559562 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":76,"threads_started":1}
I20260926 20:14:19.704888 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:19.704427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:45355)
0926 20:14:19.704730 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:19.704793 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":74,"threads_started":1}
I20260926 20:14:19.898710 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:19.898241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:47335)
0926 20:14:19.898545 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:19.898610 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":103,"threads_started":1}
I20260926 20:14:20.561059 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:20.560640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:41487)
0926 20:14:20.560902 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:20.560962 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260926 20:14:20.706214 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:20.705740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:41271)
0926 20:14:20.706059 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:20.706115 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":131,"threads_started":1}
I20260926 20:14:20.899981 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:20.899536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:60343)
0926 20:14:20.899819 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:20.899882 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260926 20:14:21.562248 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:21.561816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:59767)
0926 20:14:21.562091 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:21.562152 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":82,"threads_started":1}
I20260926 20:14:21.707443 8315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:21.707013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:50543)
0926 20:14:21.707261 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:21.707347 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260926 20:14:21.901162 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:21.900747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:37765)
0926 20:14:21.901015 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:21.901068 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
I20260926 20:14:22.563532 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:22.563069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:51121)
0926 20:14:22.563367 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:22.563431 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":77,"threads_started":1}
I20260926 20:14:22.708902 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:22.708434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:36189)
0926 20:14:22.708722 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:22.708794 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260926 20:14:22.902428 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:22.902011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:36563)
0926 20:14:22.902280 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:22.902337 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":95,"threads_started":1}
I20260926 20:14:23.564954 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:23.564521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:54433)
0926 20:14:23.564778 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:23.564857 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":95,"threads_started":1}
I20260926 20:14:23.710354 8321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:23.709793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:43635)
0926 20:14:23.710142 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:23.710211 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":93,"threads_started":1}
I20260926 20:14:23.903669 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:23.903209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:38517)
0926 20:14:23.903497 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:23.903573 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":72,"threads_started":1}
I20260926 20:14:24.278682 8323 raft_consensus.cc:482] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:14:24.278820 8323 raft_consensus.cc:504] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ab5b58d10b4417c91f6271bc3a3d452" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 39265 } } peers { permanent_uuid: "e2671a5362ec44c69479de3428f9977a" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34853 } } peers { permanent_uuid: "01868c6c2ec4413d87294a208538e121" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45491 } }
I20260926 20:14:24.279237 8323 leader_election.cc:288] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265), e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853)
W20260926 20:14:24.280057 7896 leader_election.cc:341] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2671a5362ec44c69479de3428f9977a (127.23.50.131:34853): Not found: Tablet not found: 30166f22c0b343c99c892e007ef535a8
W20260926 20:14:24.280258 7894 leader_election.cc:341] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ab5b58d10b4417c91f6271bc3a3d452 (127.23.50.130:39265): Not found: Tablet not found: 30166f22c0b343c99c892e007ef535a8
I20260926 20:14:24.280364 7894 leader_election.cc:302] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 01868c6c2ec4413d87294a208538e121; no voters: 3ab5b58d10b4417c91f6271bc3a3d452, e2671a5362ec44c69479de3428f9977a
I20260926 20:14:24.280557 8323 raft_consensus.cc:2726] T 30166f22c0b343c99c892e007ef535a8 P 01868c6c2ec4413d87294a208538e121 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 20:14:24.566174 8324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:24.565699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:42507)
0926 20:14:24.566017 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:24.566073 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":88,"threads_started":1}
I20260926 20:14:24.711637 8325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:24.711194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:55369)
0926 20:14:24.711478 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:24.711536 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
I20260926 20:14:24.905107 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:24.904636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:43353)
0926 20:14:24.904938 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:24.905006 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":95,"threads_started":1}
I20260926 20:14:25.567811 8327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:25.567336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.129:42741)
0926 20:14:25.567664 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:25.567717 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":104,"threads_started":1}
I20260926 20:14:25.713037 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:25.712602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.130:41881)
0926 20:14:25.712881 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:25.712939 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":86,"threads_started":1}
I20260926 20:14:25.897972 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 7837
I20260926 20:14:25.904356 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 8010
I20260926 20:14:25.906759 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:25.906323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:42405 (local address 127.23.50.131:50381)
0926 20:14:25.906608 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:25.906660 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.190:42405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
I20260926 20:14:25.910472 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 8144
2026-09-26T20:14:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59470 ms)
[----------] 30 tests from SecurityITest (244682 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:25 dist-test-slave-7034 krb5kdc[8332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:25 dist-test-slave-7034 krb5kdc[8332](info): set up 2 sockets
Sep 26 20:14:25 dist-test-slave-7034 krb5kdc[8332](info): commencing operation
krb5kdc: starting...
W20260926 20:14:27.030977 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:27 dist-test-slave-7034 krb5kdc[8332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:14:27.157622 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:41981
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:41981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:27.272953 8345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:27.273288 8345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:27.273370 8345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:27.277555 8345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:27.277673 8345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:27.277717 8345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:27.277756 8345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:27.277793 8345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:27.283392 8345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:41981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:41981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:27.284585 8345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:27.285672 8345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:27.294703 8353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:27.294694 8350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:27.294752 8345 server_base.cc:1034] running on GCE node
W20260926 20:14:27.294694 8351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:27.295728 8345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:27.296871 8345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:27.298025 8345 hybrid_clock.cc:648] HybridClock initialized: now 1790453667298007 us; error 30 us; skew 500 ppm
Sep 26 20:14:27 dist-test-slave-7034 krb5kdc[8332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453667, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:27.300954 8345 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:14:27.301966 8345 webserver.cc:460] Webserver started at http://127.23.50.190:46847/ using document root <none> and password file <none>
I20260926 20:14:27.302546 8345 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:27.302610 8345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:27.302887 8345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:27.304996 8345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "fb66b0fcb00c4a4782a8df3eaec498c0"
format_stamp: "Formatted at 2026-09-26 20:14:27 on dist-test-slave-7034"
server_key: "8fa565a178aec18fccc61e030261739b"
server_key_iv: "7c80ee6af6c8250f7615543b0c6ebcbf"
server_key_version: "encryptionkey@0"
I20260926 20:14:27.305666 8345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "fb66b0fcb00c4a4782a8df3eaec498c0"
format_stamp: "Formatted at 2026-09-26 20:14:27 on dist-test-slave-7034"
server_key: "8fa565a178aec18fccc61e030261739b"
server_key_iv: "7c80ee6af6c8250f7615543b0c6ebcbf"
server_key_version: "encryptionkey@0"
I20260926 20:14:27.309726 8345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 20:14:27.312191 8360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:27.313463 8345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:14:27.313593 8345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "fb66b0fcb00c4a4782a8df3eaec498c0"
format_stamp: "Formatted at 2026-09-26 20:14:27 on dist-test-slave-7034"
server_key: "8fa565a178aec18fccc61e030261739b"
server_key_iv: "7c80ee6af6c8250f7615543b0c6ebcbf"
server_key_version: "encryptionkey@0"
I20260926 20:14:27.313704 8345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:27.331146 8345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:27.334343 8345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:27.334566 8345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:27.342254 8345 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:41981
I20260926 20:14:27.342288 8412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:41981 every 8 connection(s)
I20260926 20:14:27.343266 8345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:27.344554 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 8345
I20260926 20:14:27.344712 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:27.345038 23754 external_mini_cluster.cc:1428] Setting key a58f4f8b5284eba5e6ec3429284b59b1
I20260926 20:14:27.346702 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:27.352353 8413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0: Bootstrap starting.
Sep 26 20:14:27 dist-test-slave-7034 krb5kdc[8332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:27.354982 8413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:27.355922 8413 log.cc:824] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:27.358477 8413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0: No bootstrap required, opened a new log
I20260926 20:14:27.359781 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:27.346550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.23.50.190:41981)
0926 20:14:27.347025 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:27.347040 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:27.347120 (+ 80us) server_negotiation.cc:407] Connection header received
0926 20:14:27.347768 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:27.347783 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:27.348138 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:27.348482 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:27.349314 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:27.350401 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:27.351272 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:27.351748 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:27.351927 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:27.354733 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:27.354738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:27.354753 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:27.354781 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:27.356791 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:27.357334 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:27.357339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:27.357343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:27.357456 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:27.357749 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:27.357752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:27.357753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:27.358214 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:27.358376 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:27.358703 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 20:14:27.359034 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":151,"threads_started":1}
I20260926 20:14:27.362584 8413 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb66b0fcb00c4a4782a8df3eaec498c0" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41981 } }
I20260926 20:14:27.362807 8413 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:27.362897 8413 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb66b0fcb00c4a4782a8df3eaec498c0, State: Initialized, Role: FOLLOWER
I20260926 20:14:27.363353 8413 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb66b0fcb00c4a4782a8df3eaec498c0" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41981 } }
I20260926 20:14:27.363499 8413 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:27.363565 8413 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:27.363682 8413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:27.364598 8413 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb66b0fcb00c4a4782a8df3eaec498c0" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41981 } }
I20260926 20:14:27.364971 8413 leader_election.cc:302] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb66b0fcb00c4a4782a8df3eaec498c0; no voters:
I20260926 20:14:27.365316 8413 leader_election.cc:288] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:27.365398 8418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:27.365583 8418 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [term 1 LEADER]: Becoming Leader. State: Replica: fb66b0fcb00c4a4782a8df3eaec498c0, State: Running, Role: LEADER
I20260926 20:14:27.365911 8418 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb66b0fcb00c4a4782a8df3eaec498c0" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41981 } }
I20260926 20:14:27.366639 8413 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:27.367707 8420 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb66b0fcb00c4a4782a8df3eaec498c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb66b0fcb00c4a4782a8df3eaec498c0" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41981 } } }
I20260926 20:14:27.367841 8420 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:27.368047 8419 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb66b0fcb00c4a4782a8df3eaec498c0. Latest consensus state: current_term: 1 leader_uuid: "fb66b0fcb00c4a4782a8df3eaec498c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb66b0fcb00c4a4782a8df3eaec498c0" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41981 } } }
I20260926 20:14:27.368181 8419 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0 [sys.catalog]: This master's current role is: LEADER
W20260926 20:14:27.371440 8433 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:14:27.371505 8433 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:14:27.371589 8429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:27.372246 8429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:27.378007 8429 catalog_manager.cc:1347] Generated new cluster ID: 1fb6fa4cad7e428798c2b273ba30964c
I20260926 20:14:27.378088 8429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:27.391392 8429 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:27.392346 8429 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:27.404229 8429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0: Generated new TSK 0
I20260926 20:14:27.404965 8429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:14:27.436790 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:41981
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:27.570624 8439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:27.570922 8439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:27.570966 8439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:27.575042 8439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:27.575106 8439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:27.575166 8439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:14:27.580578 8439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:41981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:27.581650 8439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:27.582664 8439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:27.592660 8447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:27.592893 8439 server_base.cc:1034] running on GCE node
W20260926 20:14:27.592649 8444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:27.592645 8445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:27.593603 8439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:27.594349 8439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:27.595543 8439 hybrid_clock.cc:648] HybridClock initialized: now 1790453667595525 us; error 32 us; skew 500 ppm
Sep 26 20:14:27 dist-test-slave-7034 krb5kdc[8332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453667, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:27.598433 8439 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:14:27.599432 8439 webserver.cc:460] Webserver started at http://127.23.50.129:34067/ using document root <none> and password file <none>
I20260926 20:14:27.600064 8439 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:27.600167 8439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:27.600428 8439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:27.602782 8439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "dc0ec16fc86f4a96a656cfe524e9dacb"
format_stamp: "Formatted at 2026-09-26 20:14:27 on dist-test-slave-7034"
server_key: "c22a55bb90408f617989f0cce92b9820"
server_key_iv: "05f8cbbbc96a838a18a29c0ca52f37cf"
server_key_version: "encryptionkey@0"
I20260926 20:14:27.603549 8439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "dc0ec16fc86f4a96a656cfe524e9dacb"
format_stamp: "Formatted at 2026-09-26 20:14:27 on dist-test-slave-7034"
server_key: "c22a55bb90408f617989f0cce92b9820"
server_key_iv: "05f8cbbbc96a838a18a29c0ca52f37cf"
server_key_version: "encryptionkey@0"
I20260926 20:14:27.607815 8439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:14:27.610913 8454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:27.612293 8439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:14:27.612458 8439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "dc0ec16fc86f4a96a656cfe524e9dacb"
format_stamp: "Formatted at 2026-09-26 20:14:27 on dist-test-slave-7034"
server_key: "c22a55bb90408f617989f0cce92b9820"
server_key_iv: "05f8cbbbc96a838a18a29c0ca52f37cf"
server_key_version: "encryptionkey@0"
I20260926 20:14:27.612596 8439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:27.639722 8439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:27.643038 8439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:27.643275 8439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:27.643925 8439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:27.644829 8439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:27.644909 8439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:27.644992 8439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:27.645041 8439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:27.655042 8439 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:39659
I20260926 20:14:27.655071 8567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:39659 every 8 connection(s)
I20260926 20:14:27.656067 8439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:14:27.664677 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 8439
I20260926 20:14:27.664829 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:14:27.665155 23754 external_mini_cluster.cc:1428] Setting key e8007f91ba6aa54b53a3dae6c301b20a
Sep 26 20:14:27 dist-test-slave-7034 krb5kdc[8332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453667, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:27.671396 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:27.658156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:45943 (local address 127.23.50.190:41981)
0926 20:14:27.658339 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:27.658344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:27.659200 (+ 856us) server_negotiation.cc:407] Connection header received
0926 20:14:27.660563 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:27.660566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:27.660635 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:27.660720 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:27.661953 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:27.662502 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:27.663656 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:27.663919 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:27.664012 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:27.667230 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:27.667236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:27.667238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:27.667273 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:27.669051 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:27.669556 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:27.669561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:27.669564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:27.669661 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:27.670121 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:27.670124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:27.670125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:27.670307 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:27.670387 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:27.671079 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 20:14:27.671190 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:14:27.672601 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:27.658551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41981 (local address 127.23.50.129:45943)
0926 20:14:27.659028 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:27.659080 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:27.660332 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:27.660889 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:27.660896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:27.661255 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:27.661770 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:27.661784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:27.662620 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:27.662622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:27.663513 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:27.663522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:27.664084 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:27.664087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:27.664255 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:27.665017 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:14:27.665041 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:27.666958 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:27.669180 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:27.669184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:27.669197 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:27.669454 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:27.669791 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:27.669797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:27.669802 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:27.670020 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:27.670418 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:27.670427 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:27.670785 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 20:14:27.671019 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 20:14:27.671413 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":117,"threads_started":1}
I20260926 20:14:27.673964 8568 heartbeater.cc:347] Connected to a master server at 127.23.50.190:41981
I20260926 20:14:27.674263 8568 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:27.674903 8568 heartbeater.cc:511] Master 127.23.50.190:41981 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:14:27.676661 8377 ts_manager.cc:194] Registered new tserver with Master: dc0ec16fc86f4a96a656cfe524e9dacb (127.23.50.129:39659)
I20260926 20:14:27.677829 8377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:45943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:14:27.693490 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:41981
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:27.807140 8573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:27.807406 8573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:27.807451 8573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:27.811672 8573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:27.811756 8573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:27.811885 8573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:14:27.817557 8573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:41981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:27.818723 8573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:27.819768 8573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:27.829550 8581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:27.829552 8578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:27.829607 8573 server_base.cc:1034] running on GCE node
W20260926 20:14:27.829547 8579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:27.830262 8573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:27.830904 8573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:27.832134 8573 hybrid_clock.cc:648] HybridClock initialized: now 1790453667832112 us; error 32 us; skew 500 ppm
Sep 26 20:14:27 dist-test-slave-7034 krb5kdc[8332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453667, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:27.835073 8573 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:14:27.836005 8573 webserver.cc:460] Webserver started at http://127.23.50.130:39051/ using document root <none> and password file <none>
I20260926 20:14:27.836930 8573 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:27.837044 8573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:27.837426 8573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:27.839954 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "904099d002f847c483b834c57eeca29f"
format_stamp: "Formatted at 2026-09-26 20:14:27 on dist-test-slave-7034"
server_key: "eea2e684cee0ef39aba1c26ee480934d"
server_key_iv: "c6f6a73823820da2a82c4ef54e37fc36"
server_key_version: "encryptionkey@0"
I20260926 20:14:27.840605 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "904099d002f847c483b834c57eeca29f"
format_stamp: "Formatted at 2026-09-26 20:14:27 on dist-test-slave-7034"
server_key: "eea2e684cee0ef39aba1c26ee480934d"
server_key_iv: "c6f6a73823820da2a82c4ef54e37fc36"
server_key_version: "encryptionkey@0"
I20260926 20:14:27.844607 8573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:14:27.847167 8588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:27.848487 8573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:14:27.848639 8573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "904099d002f847c483b834c57eeca29f"
format_stamp: "Formatted at 2026-09-26 20:14:27 on dist-test-slave-7034"
server_key: "eea2e684cee0ef39aba1c26ee480934d"
server_key_iv: "c6f6a73823820da2a82c4ef54e37fc36"
server_key_version: "encryptionkey@0"
I20260926 20:14:27.848754 8573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:27.876732 8573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:27.879835 8573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:27.880067 8573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:27.880676 8573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:27.881510 8573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:27.881577 8573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:27.881645 8573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:27.881692 8573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:27.890717 8573 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:41409
I20260926 20:14:27.890766 8701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:41409 every 8 connection(s)
I20260926 20:14:27.891662 8573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:14:27.900563 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 8573
I20260926 20:14:27.900727 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:14:27.901109 23754 external_mini_cluster.cc:1428] Setting key c488ccaee4cac513818be844ceaab967
Sep 26 20:14:27 dist-test-slave-7034 krb5kdc[8332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453667, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:27.906186 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:27.893477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:59955 (local address 127.23.50.190:41981)
0926 20:14:27.893630 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:27.893634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:27.894482 (+ 848us) server_negotiation.cc:407] Connection header received
0926 20:14:27.895897 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:27.895900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:27.895960 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:27.896071 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:27.897246 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:27.897788 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:27.898981 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:27.899216 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:27.899329 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:27.902143 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:27.902146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:27.902148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:27.902173 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:27.903741 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:27.904457 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:27.904462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:27.904465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:27.904569 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:27.904909 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:27.904913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:27.904916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:27.905172 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:27.905282 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:27.905966 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 20:14:27.906069 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 20:14:27.907493 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:27.893835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41981 (local address 127.23.50.130:59955)
0926 20:14:27.894327 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:27.894373 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:27.895656 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:27.896261 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:27.896268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:27.896619 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:27.897094 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:27.897107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:27.897942 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:27.897950 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:27.898845 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:27.898856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:27.899372 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:27.899376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:27.899548 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:27.900160 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:14:27.900177 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:27.901969 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:27.903886 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:27.903892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:27.903908 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:27.904336 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:27.904673 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:27.904676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:27.904677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:27.904815 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:27.905297 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:27.905306 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:27.905641 (+ 335us) client_negotiation.cc:769] Sending connection context
0926 20:14:27.905890 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 20:14:27.906269 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260926 20:14:27.908780 8702 heartbeater.cc:347] Connected to a master server at 127.23.50.190:41981
I20260926 20:14:27.909062 8702 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:27.909685 8702 heartbeater.cc:511] Master 127.23.50.190:41981 requested a full tablet report, sending...
I20260926 20:14:27.910939 8377 ts_manager.cc:194] Registered new tserver with Master: 904099d002f847c483b834c57eeca29f (127.23.50.130:41409)
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:14:27.911621 8377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:59955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:14:27.930150 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:41981
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:40893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:28.041455 8707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:28.041709 8707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:28.041752 8707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:28.045783 8707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:28.045902 8707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:28.046026 8707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:14:28.051518 8707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:40893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:41981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:28.052656 8707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:28.053758 8707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:28.063669 8712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:28.063668 8715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:28.063669 8713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:28.063990 8707 server_base.cc:1034] running on GCE node
I20260926 20:14:28.064333 8707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:28.064960 8707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:28.066121 8707 hybrid_clock.cc:648] HybridClock initialized: now 1790453668066107 us; error 27 us; skew 500 ppm
Sep 26 20:14:28 dist-test-slave-7034 krb5kdc[8332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453668, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:28.075487 8707 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:14:28.076442 8707 webserver.cc:460] Webserver started at http://127.23.50.131:34477/ using document root <none> and password file <none>
I20260926 20:14:28.077018 8707 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:28.077080 8707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:28.077303 8707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:28.079407 8707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "f81437fc9d974168b5ef26a122ed64a2"
format_stamp: "Formatted at 2026-09-26 20:14:28 on dist-test-slave-7034"
server_key: "2097cf4f9205ab609a24aeb1b4b393c0"
server_key_iv: "1e79673f2b87de397cd67000d75e732c"
server_key_version: "encryptionkey@0"
I20260926 20:14:28.079967 8707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "f81437fc9d974168b5ef26a122ed64a2"
format_stamp: "Formatted at 2026-09-26 20:14:28 on dist-test-slave-7034"
server_key: "2097cf4f9205ab609a24aeb1b4b393c0"
server_key_iv: "1e79673f2b87de397cd67000d75e732c"
server_key_version: "encryptionkey@0"
I20260926 20:14:28.083715 8707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260926 20:14:28.086051 8722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:28.087409 8707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 20:14:28.087509 8707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "f81437fc9d974168b5ef26a122ed64a2"
format_stamp: "Formatted at 2026-09-26 20:14:28 on dist-test-slave-7034"
server_key: "2097cf4f9205ab609a24aeb1b4b393c0"
server_key_iv: "1e79673f2b87de397cd67000d75e732c"
server_key_version: "encryptionkey@0"
I20260926 20:14:28.087594 8707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:28.122701 8707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:28.126139 8707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:28.126298 8707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:28.126915 8707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:28.127749 8707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:28.127827 8707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:28.127914 8707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:28.127961 8707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:28.137038 8707 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:42979
I20260926 20:14:28.137061 8835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:42979 every 8 connection(s)
I20260926 20:14:28.138093 8707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:14:28.146845 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 8707
I20260926 20:14:28.146984 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:14:28.147339 23754 external_mini_cluster.cc:1428] Setting key 0abde565b82f814ab00e849b9e99b9ea
Sep 26 20:14:28 dist-test-slave-7034 krb5kdc[8332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453668, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:28.152357 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.139984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:42739 (local address 127.23.50.190:41981)
0926 20:14:28.140127 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:28.140131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:28.140919 (+ 788us) server_negotiation.cc:407] Connection header received
0926 20:14:28.142363 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:28.142367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:28.142461 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:28.142558 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:28.143718 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.144262 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.145417 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.145644 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.145758 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.148560 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:28.148563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:28.148565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:28.148590 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:28.150473 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:28.150960 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:28.150963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:28.150964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:28.151042 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:28.151349 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:28.151351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:28.151352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:28.151517 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:28.151592 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:28.152119 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 20:14:28.152232 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:14:28.153271 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.140320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:41981 (local address 127.23.50.131:42739)
0926 20:14:28.140755 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:28.140806 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:28.142128 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:28.142740 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:28.142747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:28.143094 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:28.143564 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:28.143577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.144399 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.144403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:28.145282 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:28.145291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.145806 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.145811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:28.145996 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.146604 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:14:28.146630 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:28.148390 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:28.150580 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:28.150585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:28.150596 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:28.150856 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:28.151128 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:28.151130 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:28.151131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:28.151269 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:28.151606 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:28.151611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:28.151851 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 20:14:28.152008 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 20:14:28.152359 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260926 20:14:28.154429 8836 heartbeater.cc:347] Connected to a master server at 127.23.50.190:41981
I20260926 20:14:28.154673 8836 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:28.155172 8836 heartbeater.cc:511] Master 127.23.50.190:41981 requested a full tablet report, sending...
I20260926 20:14:28.156263 8377 ts_manager.cc:194] Registered new tserver with Master: f81437fc9d974168b5ef26a122ed64a2 (127.23.50.131:42979)
I20260926 20:14:28.156853 8377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:42739
I20260926 20:14:28.161710 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:14:28.171563 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.163802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.23.50.190:41981)
0926 20:14:28.163931 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:28.163936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:28.164205 (+ 269us) server_negotiation.cc:407] Connection header received
0926 20:14:28.164366 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:28.164369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:28.164422 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:28.164505 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:28.165217 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.165738 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.166751 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.166993 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.167106 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.168558 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:28.168562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:28.168564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:28.168589 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:28.169782 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:28.170262 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:28.170267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:28.170269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:28.170359 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:28.170637 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:28.170641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:28.170643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:28.170879 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:28.171026 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:28.171278 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 20:14:28.171403 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 20:14:28.174470 8377 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:14:28.177441 8377 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:14:28.192199 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.186299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:39659 (local address 127.0.0.1:38360)
0926 20:14:28.186703 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:28.186758 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:28.186897 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:28.187691 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:28.187698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:28.187722 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:28.187865 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:28.187872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.189166 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.189170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:28.190526 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:28.190537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.191670 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.191674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:28.191776 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.191810 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:14:28.191930 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 20:14:28.192053 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
I20260926 20:14:28.192212 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.186460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.23.50.129:39659)
0926 20:14:28.187149 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:28.187160 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:28.187190 (+ 30us) server_negotiation.cc:407] Connection header received
0926 20:14:28.187276 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:28.187282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:28.187517 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:28.187680 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:28.188018 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.189043 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.190666 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.191557 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.191665 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.191818 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:28.191965 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 20:14:28.192061 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":127,"threads_started":1}
I20260926 20:14:28.192869 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.186771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:41409 (local address 127.0.0.1:39644)
0926 20:14:28.187369 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:28.187383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:28.187464 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:28.188469 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:28.188472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:28.188490 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:28.188588 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:28.188594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.189922 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.189925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:28.190972 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:28.190980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.192447 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.192450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:28.192538 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.192560 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:14:28.192643 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:14:28.192714 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":86,"threads_started":1}
I20260926 20:14:28.193050 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.187062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:42979 (local address 127.0.0.1:45606)
0926 20:14:28.187904 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:28.187919 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:28.188000 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:28.188631 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:28.188634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:28.188654 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:28.188756 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:28.188762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.190110 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.190114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:28.191321 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:28.191332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.192623 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.192629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:28.192712 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.192731 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 20:14:28.192803 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 20:14:28.192857 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":97,"threads_started":1}
I20260926 20:14:28.193305 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.187663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39644 (local address 127.23.50.130:41409)
0926 20:14:28.187985 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:28.187997 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:28.188020 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:14:28.188120 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:28.188125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:28.188312 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:28.188470 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:28.188688 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.189748 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.191174 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.192114 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.192226 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.192382 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:28.193069 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 20:14:28.193171 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":119,"threads_started":1}
I20260926 20:14:28.195672 8502 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8f67ba8102477083d44c694e5c4f46 (DEFAULT_TABLE table=test-table [id=e29b97cbef7a4f2cb825ecdfb4036b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:28.195971 8636 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8f67ba8102477083d44c694e5c4f46 (DEFAULT_TABLE table=test-table [id=e29b97cbef7a4f2cb825ecdfb4036b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:28.196931 8502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8f67ba8102477083d44c694e5c4f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:28.196931 8636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8f67ba8102477083d44c694e5c4f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:28.198321 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.187436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.23.50.131:42979)
0926 20:14:28.188112 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:28.188122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:28.188141 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:14:28.188203 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:28.188207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:28.188341 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:28.188470 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:28.188892 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.189951 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.191443 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.192241 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.192440 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.192567 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:28.192941 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 20:14:28.197900 (+ 4959us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":201,"threads_started":1}
I20260926 20:14:28.200933 8770 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8f67ba8102477083d44c694e5c4f46 (DEFAULT_TABLE table=test-table [id=e29b97cbef7a4f2cb825ecdfb4036b34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:28.202049 8770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8f67ba8102477083d44c694e5c4f46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:28.204622 8854 tablet_bootstrap.cc:492] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f: Bootstrap starting.
I20260926 20:14:28.205071 8853 tablet_bootstrap.cc:492] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb: Bootstrap starting.
I20260926 20:14:28.207649 8854 tablet_bootstrap.cc:654] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:28.208110 8853 tablet_bootstrap.cc:654] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:28.209079 8854 log.cc:824] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:28.209090 8853 log.cc:824] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:28.209923 8857 tablet_bootstrap.cc:492] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2: Bootstrap starting.
I20260926 20:14:28.211465 8854 tablet_bootstrap.cc:492] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f: No bootstrap required, opened a new log
I20260926 20:14:28.211699 8854 ts_tablet_manager.cc:1397] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 20:14:28.212285 8857 tablet_bootstrap.cc:654] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:28.213490 8857 log.cc:824] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:28.215236 8854 raft_consensus.cc:348] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 39659 } } peers { permanent_uuid: "f81437fc9d974168b5ef26a122ed64a2" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42979 } } peers { permanent_uuid: "904099d002f847c483b834c57eeca29f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41409 } }
I20260926 20:14:28.215543 8854 raft_consensus.cc:374] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:28.215617 8854 raft_consensus.cc:729] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 904099d002f847c483b834c57eeca29f, State: Initialized, Role: FOLLOWER
I20260926 20:14:28.215665 8857 tablet_bootstrap.cc:492] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2: No bootstrap required, opened a new log
I20260926 20:14:28.215873 8857 ts_tablet_manager.cc:1397] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 20:14:28.216113 8854 consensus_queue.cc:260] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 39659 } } peers { permanent_uuid: "f81437fc9d974168b5ef26a122ed64a2" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42979 } } peers { permanent_uuid: "904099d002f847c483b834c57eeca29f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41409 } }
I20260926 20:14:28.215485 8853 tablet_bootstrap.cc:492] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb: No bootstrap required, opened a new log
I20260926 20:14:28.216569 8853 ts_tablet_manager.cc:1397] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260926 20:14:28.217001 8702 heartbeater.cc:503] Master 127.23.50.190:41981 was elected leader, sending a full tablet report...
I20260926 20:14:28.217433 8854 ts_tablet_manager.cc:1428] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:14:28.220408 8853 raft_consensus.cc:348] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 39659 } } peers { permanent_uuid: "f81437fc9d974168b5ef26a122ed64a2" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42979 } } peers { permanent_uuid: "904099d002f847c483b834c57eeca29f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41409 } }
I20260926 20:14:28.220721 8853 raft_consensus.cc:374] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:28.220788 8853 raft_consensus.cc:729] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc0ec16fc86f4a96a656cfe524e9dacb, State: Initialized, Role: FOLLOWER
I20260926 20:14:28.221278 8853 consensus_queue.cc:260] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 39659 } } peers { permanent_uuid: "f81437fc9d974168b5ef26a122ed64a2" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42979 } } peers { permanent_uuid: "904099d002f847c483b834c57eeca29f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41409 } }
I20260926 20:14:28.221778 8857 raft_consensus.cc:348] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 39659 } } peers { permanent_uuid: "f81437fc9d974168b5ef26a122ed64a2" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42979 } } peers { permanent_uuid: "904099d002f847c483b834c57eeca29f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41409 } }
I20260926 20:14:28.222050 8568 heartbeater.cc:503] Master 127.23.50.190:41981 was elected leader, sending a full tablet report...
I20260926 20:14:28.222138 8857 raft_consensus.cc:374] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:28.222210 8857 raft_consensus.cc:729] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f81437fc9d974168b5ef26a122ed64a2, State: Initialized, Role: FOLLOWER
I20260926 20:14:28.222473 8853 ts_tablet_manager.cc:1428] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260926 20:14:28.222754 8857 consensus_queue.cc:260] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 39659 } } peers { permanent_uuid: "f81437fc9d974168b5ef26a122ed64a2" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42979 } } peers { permanent_uuid: "904099d002f847c483b834c57eeca29f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41409 } }
I20260926 20:14:28.223543 8836 heartbeater.cc:503] Master 127.23.50.190:41981 was elected leader, sending a full tablet report...
I20260926 20:14:28.224006 8857 ts_tablet_manager.cc:1428] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
W20260926 20:14:28.389134 8837 tablet.cc:2369] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:14:28.392769 8703 tablet.cc:2369] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:14:28.407572 8569 tablet.cc:2369] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:14:28.510769 8861 raft_consensus.cc:482] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:14:28.511137 8861 raft_consensus.cc:504] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 39659 } } peers { permanent_uuid: "f81437fc9d974168b5ef26a122ed64a2" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42979 } } peers { permanent_uuid: "904099d002f847c483b834c57eeca29f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41409 } }
I20260926 20:14:28.512387 8861 leader_election.cc:288] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc0ec16fc86f4a96a656cfe524e9dacb (127.23.50.129:39659), 904099d002f847c483b834c57eeca29f (127.23.50.130:41409)
I20260926 20:14:28.517130 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.512650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:41409 (local address 127.23.50.131:51913)
0926 20:14:28.512761 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:28.512775 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:28.512877 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:28.513199 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:28.513202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:28.513224 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:28.513369 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:28.513378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.514405 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.514408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:28.515667 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:28.515678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.516596 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.516600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:28.516816 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.516834 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 20:14:28.516901 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 20:14:28.516987 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":98}
I20260926 20:14:28.517128 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.512759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:51913 (local address 127.23.50.130:41409)
0926 20:14:28.512917 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:28.512924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:28.512945 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:14:28.513010 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:28.513013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:28.513066 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:28.513164 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:28.513494 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.514284 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.515808 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.516445 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.516577 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.516631 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:28.516932 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 20:14:28.517010 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 20:14:28.517166 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.512897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:54067 (local address 127.23.50.129:39659)
0926 20:14:28.513045 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:28.513050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:28.513321 (+ 271us) server_negotiation.cc:407] Connection header received
0926 20:14:28.513440 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:28.513445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:28.513498 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:28.513577 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:28.513896 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.514548 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.515897 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.516488 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.516592 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.516631 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:28.516978 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 20:14:28.517047 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:14:28.517175 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.512658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:39659 (local address 127.23.50.131:54067)
0926 20:14:28.513205 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:28.513218 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:28.513314 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:28.513626 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:28.513628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:28.513645 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:28.513751 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:28.513756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.514647 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.514650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:28.515787 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:28.515797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.516596 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.516600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:28.516752 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.516816 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 20:14:28.516962 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 20:14:28.517063 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":161,"thread_start_us":143,"threads_started":1}
I20260926 20:14:28.517925 8656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8f67ba8102477083d44c694e5c4f46" candidate_uuid: "f81437fc9d974168b5ef26a122ed64a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "904099d002f847c483b834c57eeca29f" is_pre_election: true
I20260926 20:14:28.517980 8522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8f67ba8102477083d44c694e5c4f46" candidate_uuid: "f81437fc9d974168b5ef26a122ed64a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb" is_pre_election: true
I20260926 20:14:28.518249 8656 raft_consensus.cc:2454] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f81437fc9d974168b5ef26a122ed64a2 in term 0.
I20260926 20:14:28.518285 8522 raft_consensus.cc:2454] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f81437fc9d974168b5ef26a122ed64a2 in term 0.
I20260926 20:14:28.518787 8726 leader_election.cc:302] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 904099d002f847c483b834c57eeca29f, dc0ec16fc86f4a96a656cfe524e9dacb, f81437fc9d974168b5ef26a122ed64a2; no voters:
I20260926 20:14:28.519078 8861 raft_consensus.cc:2781] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:14:28.519165 8861 raft_consensus.cc:482] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:14:28.519208 8861 raft_consensus.cc:3037] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:28.520223 8861 raft_consensus.cc:504] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 39659 } } peers { permanent_uuid: "f81437fc9d974168b5ef26a122ed64a2" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42979 } } peers { permanent_uuid: "904099d002f847c483b834c57eeca29f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41409 } }
I20260926 20:14:28.520581 8861 leader_election.cc:288] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [CANDIDATE]: Term 1 election: Requested vote from peers dc0ec16fc86f4a96a656cfe524e9dacb (127.23.50.129:39659), 904099d002f847c483b834c57eeca29f (127.23.50.130:41409)
I20260926 20:14:28.520877 8522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8f67ba8102477083d44c694e5c4f46" candidate_uuid: "f81437fc9d974168b5ef26a122ed64a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb"
I20260926 20:14:28.521016 8656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8f67ba8102477083d44c694e5c4f46" candidate_uuid: "f81437fc9d974168b5ef26a122ed64a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "904099d002f847c483b834c57eeca29f"
I20260926 20:14:28.521052 8522 raft_consensus.cc:3037] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:28.521139 8656 raft_consensus.cc:3037] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:28.522559 8522 raft_consensus.cc:2454] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f81437fc9d974168b5ef26a122ed64a2 in term 1.
I20260926 20:14:28.522559 8656 raft_consensus.cc:2454] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f81437fc9d974168b5ef26a122ed64a2 in term 1.
I20260926 20:14:28.522908 8726 leader_election.cc:302] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc0ec16fc86f4a96a656cfe524e9dacb, f81437fc9d974168b5ef26a122ed64a2; no voters:
I20260926 20:14:28.523075 8861 raft_consensus.cc:2781] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:28.523307 8861 raft_consensus.cc:686] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 1 LEADER]: Becoming Leader. State: Replica: f81437fc9d974168b5ef26a122ed64a2, State: Running, Role: LEADER
I20260926 20:14:28.523672 8861 consensus_queue.cc:237] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 39659 } } peers { permanent_uuid: "f81437fc9d974168b5ef26a122ed64a2" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42979 } } peers { permanent_uuid: "904099d002f847c483b834c57eeca29f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41409 } }
I20260926 20:14:28.526798 8376 catalog_manager.cc:5495] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f81437fc9d974168b5ef26a122ed64a2 (127.23.50.131). New cstate: current_term: 1 leader_uuid: "f81437fc9d974168b5ef26a122ed64a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 39659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f81437fc9d974168b5ef26a122ed64a2" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "904099d002f847c483b834c57eeca29f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41409 } health_report { overall_health: UNKNOWN } } }
I20260926 20:14:28.556803 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.553005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.23.50.131:42979)
0926 20:14:28.553183 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:28.553188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:28.553203 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:14:28.553418 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:28.553420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:28.553484 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:28.553608 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:14:28.553887 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.554401 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.555287 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.555513 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.555628 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.555948 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:14:28.556382 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:14:28.556523 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:28.556563 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 20:14:28.556621 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:14:28.562930 8522 raft_consensus.cc:1261] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb [term 1 FOLLOWER]: Refusing update from remote peer f81437fc9d974168b5ef26a122ed64a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:14:28.562917 8656 raft_consensus.cc:1261] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f [term 1 FOLLOWER]: Refusing update from remote peer f81437fc9d974168b5ef26a122ed64a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:14:28.563730 8863 consensus_queue.cc:1035] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "904099d002f847c483b834c57eeca29f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:14:28.563934 8861 consensus_queue.cc:1035] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc0ec16fc86f4a96a656cfe524e9dacb" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:14:28.573284 8868 mvcc.cc:205] Tried to move back new op lower bound from 7333698226425184256 to 7333698226275680256. Current Snapshot: MvccSnapshot[applied={T|T < 7333698226425184256}]
I20260926 20:14:28.573505 8867 mvcc.cc:205] Tried to move back new op lower bound from 7333698226425184256 to 7333698226275680256. Current Snapshot: MvccSnapshot[applied={T|T < 7333698226425184256}]
I20260926 20:14:28.581027 8377 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:14:28.581243 8377 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:14:28.583890 8377 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb66b0fcb00c4a4782a8df3eaec498c0: Sending DeleteTablet for 3 replicas of tablet 8f8f67ba8102477083d44c694e5c4f46
I20260926 20:14:28.584784 8636 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8f67ba8102477083d44c694e5c4f46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:28 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:39644
I20260926 20:14:28.584962 8770 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8f67ba8102477083d44c694e5c4f46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:28 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:45606
I20260926 20:14:28.585007 8502 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8f67ba8102477083d44c694e5c4f46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:28 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:38360
I20260926 20:14:28.585387 8876 tablet_replica.cc:330] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f: stopping tablet replica
I20260926 20:14:28.585556 8878 tablet_replica.cc:330] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb: stopping tablet replica
I20260926 20:14:28.585835 8876 raft_consensus.cc:2229] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:14:28.585897 8878 raft_consensus.cc:2229] T 8f8f67ba8102477083d44c694e5c4f46 P dc0ec16fc86f4a96a656cfe524e9dacb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:14:28.586012 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 8439
I20260926 20:14:28.587527 8876 raft_consensus.cc:2258] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:28.589331 8876 ts_tablet_manager.cc:1905] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:14:28.590056 8877 tablet_replica.cc:330] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2: stopping tablet replica
I20260926 20:14:28.590430 8877 raft_consensus.cc:2229] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:14:28.590960 8877 raft_consensus.cc:2258] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:28.592639 8877 ts_tablet_manager.cc:1905] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:14:28.593067 8876 ts_tablet_manager.cc:1918] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:14:28.593181 8876 log.cc:1196] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/8f8f67ba8102477083d44c694e5c4f46
I20260926 20:14:28.593535 8876 ts_tablet_manager.cc:1939] T 8f8f67ba8102477083d44c694e5c4f46 P 904099d002f847c483b834c57eeca29f: Deleting consensus metadata
I20260926 20:14:28.595014 8363 catalog_manager.cc:4841] TS 904099d002f847c483b834c57eeca29f (127.23.50.130:41409): tablet 8f8f67ba8102477083d44c694e5c4f46 (table test-table [id=e29b97cbef7a4f2cb825ecdfb4036b34]) successfully deleted
W20260926 20:14:28.595683 8364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:39659 (error 108)
I20260926 20:14:28.595846 8877 ts_tablet_manager.cc:1918] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:14:28.595949 8877 log.cc:1196] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/8f8f67ba8102477083d44c694e5c4f46
I20260926 20:14:28.596251 8877 ts_tablet_manager.cc:1939] T 8f8f67ba8102477083d44c694e5c4f46 P f81437fc9d974168b5ef26a122ed64a2: Deleting consensus metadata
I20260926 20:14:28.596536 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 8573
I20260926 20:14:28.598104 8364 catalog_manager.cc:4841] TS f81437fc9d974168b5ef26a122ed64a2 (127.23.50.131:42979): tablet 8f8f67ba8102477083d44c694e5c4f46 (table test-table [id=e29b97cbef7a4f2cb825ecdfb4036b34]) successfully deleted
I20260926 20:14:28.603668 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.603280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:39659 (local address 127.0.0.1:38370)
0926 20:14:28.603466 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:28.603549 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260926 20:14:28.603900 8364 catalog_manager.cc:4584] TS dc0ec16fc86f4a96a656cfe524e9dacb (127.23.50.129:39659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f8f67ba8102477083d44c694e5c4f46: Network error: Client connection negotiation failed: client connection to 127.23.50.129:39659: connect: Connection refused (error 111)
I20260926 20:14:28.604004 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 8707
I20260926 20:14:28.610162 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 8345
2026-09-26T20:14:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2695 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T20:14:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:28Z Disabled control of system clock
I20260926 20:14:28.641177 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:36283
--webserver_interface=127.23.50.190
--webserver_port=0
--builtin_ntp_servers=127.23.50.148:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:14:28.758219 8883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:28.758529 8883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:28.758567 8883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:28.762723 8883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:28.762799 8883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:28.762814 8883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:28.762831 8883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:28.762847 8883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:28.767905 8883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:36283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:28.769054 8883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:28.770488 8883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:28.779059 8891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:28.779086 8888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:28.779109 8889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:28.779578 8883 server_base.cc:1034] running on GCE node
I20260926 20:14:28.780196 8883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:28.781361 8883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:28.782564 8883 hybrid_clock.cc:648] HybridClock initialized: now 1790453668782551 us; error 27 us; skew 500 ppm
I20260926 20:14:28.784598 8883 webserver.cc:460] Webserver started at http://127.23.50.190:33765/ using document root <none> and password file <none>
I20260926 20:14:28.785311 8883 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:28.785411 8883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:28.785734 8883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:28.788053 8883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "33b86f086e5949639f3a43774b874e71"
format_stamp: "Formatted at 2026-09-26 20:14:28 on dist-test-slave-7034"
server_key: "dbbfc9c310c8e7750d983774e1453bbd"
server_key_iv: "8c7607c413744f8296c5f4f80c353558"
server_key_version: "encryptionkey@0"
I20260926 20:14:28.788793 8883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "33b86f086e5949639f3a43774b874e71"
format_stamp: "Formatted at 2026-09-26 20:14:28 on dist-test-slave-7034"
server_key: "dbbfc9c310c8e7750d983774e1453bbd"
server_key_iv: "8c7607c413744f8296c5f4f80c353558"
server_key_version: "encryptionkey@0"
I20260926 20:14:28.793094 8883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:14:28.796089 8897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:28.797627 8883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:14:28.797801 8883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "33b86f086e5949639f3a43774b874e71"
format_stamp: "Formatted at 2026-09-26 20:14:28 on dist-test-slave-7034"
server_key: "dbbfc9c310c8e7750d983774e1453bbd"
server_key_iv: "8c7607c413744f8296c5f4f80c353558"
server_key_version: "encryptionkey@0"
I20260926 20:14:28.797989 8883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:28.811393 8883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:28.812229 8883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:28.812443 8883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:28.819733 8883 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:36283
I20260926 20:14:28.819780 8949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:36283 every 8 connection(s)
I20260926 20:14:28.820745 8883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:28.824096 8950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:28.827592 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 8883
I20260926 20:14:28.827785 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:28.828061 23754 external_mini_cluster.cc:1428] Setting key f195e3e93ae2cd5f27b21d5ecb6f1197
I20260926 20:14:28.830760 8950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71: Bootstrap starting.
I20260926 20:14:28.833482 8950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:28.834427 8950 log.cc:824] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:28.836742 8950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71: No bootstrap required, opened a new log
I20260926 20:14:28.839353 8953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:28.829596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.23.50.190:36283)
0926 20:14:28.830329 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:28.830350 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:28.830393 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:14:28.831233 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:28.831257 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:28.831752 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:28.832141 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:28.832628 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.834089 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.835317 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:28.835983 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:28.836150 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:28.837102 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:28.837110 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:28.837131 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:28.837646 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:28.837719 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:28.837746 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:28.837871 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:28.838072 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 20:14:28.838360 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":103,"threads_started":1}
I20260926 20:14:28.841303 8950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b86f086e5949639f3a43774b874e71" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 36283 } }
I20260926 20:14:28.841550 8950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:28.841620 8950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b86f086e5949639f3a43774b874e71, State: Initialized, Role: FOLLOWER
I20260926 20:14:28.842082 8950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b86f086e5949639f3a43774b874e71" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 36283 } }
I20260926 20:14:28.842206 8950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:28.842255 8950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:28.842326 8950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:28.843179 8950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b86f086e5949639f3a43774b874e71" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 36283 } }
I20260926 20:14:28.843472 8950 leader_election.cc:302] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33b86f086e5949639f3a43774b874e71; no voters:
I20260926 20:14:28.843724 8950 leader_election.cc:288] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:28.843874 8955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:28.844156 8955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [term 1 LEADER]: Becoming Leader. State: Replica: 33b86f086e5949639f3a43774b874e71, State: Running, Role: LEADER
I20260926 20:14:28.844460 8955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b86f086e5949639f3a43774b874e71" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 36283 } }
I20260926 20:14:28.844934 8950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:28.846108 8956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33b86f086e5949639f3a43774b874e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b86f086e5949639f3a43774b874e71" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 36283 } } }
I20260926 20:14:28.846256 8956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:28.846591 8964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:28.847936 8957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33b86f086e5949639f3a43774b874e71. Latest consensus state: current_term: 1 leader_uuid: "33b86f086e5949639f3a43774b874e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b86f086e5949639f3a43774b874e71" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 36283 } } }
I20260926 20:14:28.848071 8957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:28.850176 8964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:28.856113 8964 catalog_manager.cc:1347] Generated new cluster ID: 6cd6bed0f45f46baa4f1636679f993d2
I20260926 20:14:28.856191 8964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:28.867651 8964 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:28.868983 8964 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:28.878229 8964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71: Generated new TSK 0
I20260926 20:14:28.879144 8964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:28.883930 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:36283
--builtin_ntp_servers=127.23.50.148:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:14:29.001540 8974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:29.001926 8974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:29.001978 8974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:29.006268 8974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:29.006336 8974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:29.006394 8974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:14:29.011448 8974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:29.012632 8974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:29.014019 8974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:14:29.023857 8974 server_base.cc:1034] running on GCE node
W20260926 20:14:29.023818 8982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:29.023836 8980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:29.023816 8979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:29.024554 8974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:29.025187 8974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:29.026353 8974 hybrid_clock.cc:648] HybridClock initialized: now 1790453669026338 us; error 30 us; skew 500 ppm
I20260926 20:14:29.028149 8974 webserver.cc:460] Webserver started at http://127.23.50.129:46557/ using document root <none> and password file <none>
I20260926 20:14:29.028743 8974 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:29.028805 8974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:29.029058 8974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:29.031260 8974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "6ee41549482449bd803064fee0403dd6"
format_stamp: "Formatted at 2026-09-26 20:14:29 on dist-test-slave-7034"
server_key: "87f62c6a0f45eb604fb0302f6973ea58"
server_key_iv: "106b2a17c477e3ff68e79618c678a580"
server_key_version: "encryptionkey@0"
I20260926 20:14:29.031823 8974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "6ee41549482449bd803064fee0403dd6"
format_stamp: "Formatted at 2026-09-26 20:14:29 on dist-test-slave-7034"
server_key: "87f62c6a0f45eb604fb0302f6973ea58"
server_key_iv: "106b2a17c477e3ff68e79618c678a580"
server_key_version: "encryptionkey@0"
I20260926 20:14:29.035691 8974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:14:29.038208 8988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:29.039593 8974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 20:14:29.039726 8974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "6ee41549482449bd803064fee0403dd6"
format_stamp: "Formatted at 2026-09-26 20:14:29 on dist-test-slave-7034"
server_key: "87f62c6a0f45eb604fb0302f6973ea58"
server_key_iv: "106b2a17c477e3ff68e79618c678a580"
server_key_version: "encryptionkey@0"
I20260926 20:14:29.039846 8974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:29.051050 8974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:29.051771 8974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:29.051980 8974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:29.052594 8974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:29.054080 8974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:29.054149 8974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:29.054224 8974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:29.054255 8974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:29.065289 8974 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:45187
I20260926 20:14:29.065315 9101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:45187 every 8 connection(s)
I20260926 20:14:29.066305 8974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:14:29.070845 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 8974
I20260926 20:14:29.071000 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:14:29.071390 23754 external_mini_cluster.cc:1428] Setting key addc0640256fc14a659a1a054359c072
I20260926 20:14:29.073462 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:36283
--builtin_ntp_servers=127.23.50.148:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:14:29.076263 8953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.068304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:50299 (local address 127.23.50.190:36283)
0926 20:14:29.068454 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:29.068463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:29.069573 (+ 1110us) server_negotiation.cc:407] Connection header received
0926 20:14:29.071144 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:29.071148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:29.071218 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:29.071332 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:29.072362 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.072907 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.074225 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.074457 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.074572 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.075746 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:29.075750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:29.075752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:29.075764 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:29.075783 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:29.075791 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:29.075869 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:29.076026 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 20:14:29.076088 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 20:14:29.077093 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.068729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:36283 (local address 127.23.50.129:50299)
0926 20:14:29.069416 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:29.069457 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:29.070875 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:29.071510 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:29.071521 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:29.071617 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:29.072224 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.072240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.073038 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.073041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.074094 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.074103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.074610 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.074615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.074786 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.075453 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:14:29.075455 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:29.075495 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:14:29.075561 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:29.075871 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:29.075883 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:29.075890 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:14:29.076025 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 20:14:29.076313 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":121,"threads_started":1}
I20260926 20:14:29.078414 9102 heartbeater.cc:347] Connected to a master server at 127.23.50.190:36283
I20260926 20:14:29.078670 9102 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:29.079190 9102 heartbeater.cc:511] Master 127.23.50.190:36283 requested a full tablet report, sending...
I20260926 20:14:29.080657 8914 ts_manager.cc:194] Registered new tserver with Master: 6ee41549482449bd803064fee0403dd6 (127.23.50.129:45187)
I20260926 20:14:29.081817 8914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.50.129:50299
W20260926 20:14:29.192160 9105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:29.192473 9105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:29.192546 9105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:29.196614 9105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:29.196705 9105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:29.196797 9105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:14:29.201819 9105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:29.203050 9105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:29.204475 9105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:29.214166 9113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:29.214438 9105 server_base.cc:1034] running on GCE node
W20260926 20:14:29.214166 9110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:29.214186 9111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:29.215075 9105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:29.215758 9105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:29.216958 9105 hybrid_clock.cc:648] HybridClock initialized: now 1790453669216923 us; error 49 us; skew 500 ppm
I20260926 20:14:29.218987 9105 webserver.cc:460] Webserver started at http://127.23.50.130:37219/ using document root <none> and password file <none>
I20260926 20:14:29.219663 9105 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:29.219762 9105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:29.220021 9105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:29.222321 9105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "087bce7ad6cb4cb6a29699b1a129b7a9"
format_stamp: "Formatted at 2026-09-26 20:14:29 on dist-test-slave-7034"
server_key: "c1b75c4904864c3b21ffb8be385b44ea"
server_key_iv: "ccbf9a114e2b21c82f130713980504e4"
server_key_version: "encryptionkey@0"
I20260926 20:14:29.222958 9105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "087bce7ad6cb4cb6a29699b1a129b7a9"
format_stamp: "Formatted at 2026-09-26 20:14:29 on dist-test-slave-7034"
server_key: "c1b75c4904864c3b21ffb8be385b44ea"
server_key_iv: "ccbf9a114e2b21c82f130713980504e4"
server_key_version: "encryptionkey@0"
I20260926 20:14:29.226975 9105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:14:29.229585 9119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:29.230985 9105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:14:29.231138 9105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "087bce7ad6cb4cb6a29699b1a129b7a9"
format_stamp: "Formatted at 2026-09-26 20:14:29 on dist-test-slave-7034"
server_key: "c1b75c4904864c3b21ffb8be385b44ea"
server_key_iv: "ccbf9a114e2b21c82f130713980504e4"
server_key_version: "encryptionkey@0"
I20260926 20:14:29.231267 9105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:29.257146 9105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:29.258003 9105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:29.258206 9105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:29.258852 9105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:29.259824 9105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:29.259893 9105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:29.259963 9105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:29.260013 9105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:29.269768 9105 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:43237
I20260926 20:14:29.269783 9232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:43237 every 8 connection(s)
I20260926 20:14:29.270875 9105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:14:29.280570 8953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.272886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:49951 (local address 127.23.50.190:36283)
0926 20:14:29.273032 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:29.273036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:29.274279 (+ 1243us) server_negotiation.cc:407] Connection header received
0926 20:14:29.275747 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:29.275751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:29.275812 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:29.275915 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:29.276841 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.277376 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.278581 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.278809 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.278922 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.280036 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:29.280039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:29.280041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:29.280054 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:29.280074 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:29.280082 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:29.280161 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:29.280356 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 20:14:29.280420 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:14:29.281093 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 9105
I20260926 20:14:29.281265 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:14:29.281543 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.273366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:36283 (local address 127.23.50.130:49951)
0926 20:14:29.274116 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:29.274165 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:29.275528 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:29.276089 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:29.276096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:29.276202 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:29.276691 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.276704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.277504 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.277507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.278448 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.278457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.278969 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.278973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.279150 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.279767 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:14:29.279769 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:29.279808 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:14:29.279872 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:29.280214 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:29.280221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:29.280226 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:14:29.280356 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 20:14:29.280640 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":132,"threads_started":1}
I20260926 20:14:29.281574 23754 external_mini_cluster.cc:1428] Setting key eb9d76632eac66110bd5929412716ec0
I20260926 20:14:29.282773 9233 heartbeater.cc:347] Connected to a master server at 127.23.50.190:36283
I20260926 20:14:29.283008 9233 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:29.283529 9233 heartbeater.cc:511] Master 127.23.50.190:36283 requested a full tablet report, sending...
I20260926 20:14:29.283798 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:36283
--builtin_ntp_servers=127.23.50.148:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:14:29.284739 8914 ts_manager.cc:194] Registered new tserver with Master: 087bce7ad6cb4cb6a29699b1a129b7a9 (127.23.50.130:43237)
I20260926 20:14:29.285622 8914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.50.130:49951
W20260926 20:14:29.408252 9236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:29.408541 9236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:29.408582 9236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:29.412819 9236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:29.412887 9236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:29.412945 9236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:14:29.418104 9236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:36283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:29.419266 9236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:29.420670 9236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:29.430341 9241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:29.430351 9244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:29.430351 9242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:29.430578 9236 server_base.cc:1034] running on GCE node
I20260926 20:14:29.431062 9236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:29.431769 9236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:29.432963 9236 hybrid_clock.cc:648] HybridClock initialized: now 1790453669432896 us; error 80 us; skew 500 ppm
I20260926 20:14:29.435189 9236 webserver.cc:460] Webserver started at http://127.23.50.131:41693/ using document root <none> and password file <none>
I20260926 20:14:29.435847 9236 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:29.435913 9236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:29.436209 9236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:29.438422 9236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "c6cb48f61f584eacbb2a2d48418fdb06"
format_stamp: "Formatted at 2026-09-26 20:14:29 on dist-test-slave-7034"
server_key: "d6f9a70bc0dd865816d1a93f69282458"
server_key_iv: "717554b4c453311edfce517b53a4399b"
server_key_version: "encryptionkey@0"
I20260926 20:14:29.438999 9236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "c6cb48f61f584eacbb2a2d48418fdb06"
format_stamp: "Formatted at 2026-09-26 20:14:29 on dist-test-slave-7034"
server_key: "d6f9a70bc0dd865816d1a93f69282458"
server_key_iv: "717554b4c453311edfce517b53a4399b"
server_key_version: "encryptionkey@0"
I20260926 20:14:29.443280 9236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:14:29.446274 9250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:29.447669 9236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 20:14:29.447839 9236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "c6cb48f61f584eacbb2a2d48418fdb06"
format_stamp: "Formatted at 2026-09-26 20:14:29 on dist-test-slave-7034"
server_key: "d6f9a70bc0dd865816d1a93f69282458"
server_key_iv: "717554b4c453311edfce517b53a4399b"
server_key_version: "encryptionkey@0"
I20260926 20:14:29.447968 9236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:29.466161 9236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:29.466872 9236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:29.467077 9236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:29.467677 9236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:29.468662 9236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:29.468734 9236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:29.468818 9236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:29.468861 9236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:29.477494 9236 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:39961
I20260926 20:14:29.477531 9363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:39961 every 8 connection(s)
I20260926 20:14:29.478434 9236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:14:29.482012 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 9236
I20260926 20:14:29.482146 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:14:29.482436 23754 external_mini_cluster.cc:1428] Setting key fcd38d21eaf7ac723cfb831543020e72
I20260926 20:14:29.488459 8953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.480331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:49249 (local address 127.23.50.190:36283)
0926 20:14:29.480501 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:29.480506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:29.481607 (+ 1101us) server_negotiation.cc:407] Connection header received
0926 20:14:29.483269 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:29.483276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:29.483357 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:29.483444 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:29.484692 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.485303 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.486579 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.486806 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.486899 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.487983 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:29.487986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:29.487988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:29.488001 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:29.488021 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:29.488030 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:29.488101 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:29.488258 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 20:14:29.488344 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:14:29.489332 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.480791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:36283 (local address 127.23.50.131:49249)
0926 20:14:29.481449 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:29.481491 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:29.483013 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:29.483639 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:29.483647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:29.483780 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:29.484489 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.484510 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.485461 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.485464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.486444 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.486455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.486960 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.486963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.487111 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.487719 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:14:29.487720 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:29.487764 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:14:29.487833 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:29.488119 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:29.488126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:29.488130 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:14:29.488246 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 20:14:29.488479 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260926 20:14:29.490535 9364 heartbeater.cc:347] Connected to a master server at 127.23.50.190:36283
I20260926 20:14:29.490789 9364 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:29.491335 9364 heartbeater.cc:511] Master 127.23.50.190:36283 requested a full tablet report, sending...
I20260926 20:14:29.492390 8914 ts_manager.cc:194] Registered new tserver with Master: c6cb48f61f584eacbb2a2d48418fdb06 (127.23.50.131:39961)
I20260926 20:14:29.493012 8914 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.50.131:49249
I20260926 20:14:29.496713 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:14:29.502758 8953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.498729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.23.50.190:36283)
0926 20:14:29.498894 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:29.498898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:29.499068 (+ 170us) server_negotiation.cc:407] Connection header received
0926 20:14:29.499216 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:29.499219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:29.499265 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:29.499385 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:29.499671 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.500189 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.501047 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.501287 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.501399 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.502330 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:29.502333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:29.502336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:29.502349 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:29.502375 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:29.502383 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:29.502465 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:29.502554 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 20:14:29.502611 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:14:29.505671 8914 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:14:29.508701 8914 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:14:29.524217 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.517809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.23.50.131:39961)
0926 20:14:29.518896 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:29.518917 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:29.518945 (+ 28us) server_negotiation.cc:407] Connection header received
0926 20:14:29.519057 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:29.519065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:29.519268 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:29.519400 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:29.519735 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.520914 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.522701 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.523431 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.523742 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.523911 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:29.523965 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 20:14:29.524064 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":144,"threads_started":1}
I20260926 20:14:29.524210 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.517675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:39961 (local address 127.0.0.1:35670)
0926 20:14:29.518571 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:29.518626 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:29.518758 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:29.519429 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:29.519435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:29.519459 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:29.519598 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.519604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.521048 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.521052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.522571 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.522582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.523572 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.523576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.523726 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.523772 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 20:14:29.523881 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 20:14:29.524005 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":72,"thread_start_us":86,"threads_started":1}
I20260926 20:14:29.524325 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.517951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.23.50.130:43237)
0926 20:14:29.519312 (+ 1361us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:29.519321 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:29.519340 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:14:29.519436 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:29.519441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:29.519616 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:29.519764 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:29.520084 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.520910 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.522619 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.523431 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.523547 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.523736 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:29.524047 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 20:14:29.524154 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1273,"thread_start_us":119,"threads_started":1}
I20260926 20:14:29.524358 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.518364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:45187 (local address 127.0.0.1:33246)
0926 20:14:29.518989 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:29.519001 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:29.519080 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:29.520113 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:29.520116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:29.520136 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:29.520228 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.520233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.521555 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.521558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.522752 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.522758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.523524 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.523527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.523629 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.524105 (+ 476us) client_negotiation.cc:769] Sending connection context
0926 20:14:29.524171 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 20:14:29.524239 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":101,"thread_start_us":82,"threads_started":1}
I20260926 20:14:29.524871 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.518516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33246 (local address 127.23.50.129:45187)
0926 20:14:29.519652 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:29.519686 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:29.519710 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:14:29.519788 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:29.519793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:29.519968 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:29.520110 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:29.520346 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.521427 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.522871 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.523397 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.523526 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.523738 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:29.524637 (+ 899us) server_negotiation.cc:299] Negotiation successful
0926 20:14:29.524744 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":95,"threads_started":1}
I20260926 20:14:29.525825 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.517675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:43237 (local address 127.0.0.1:55218)
0926 20:14:29.518715 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:29.518728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:29.518795 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:29.519882 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:29.519884 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:29.519898 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:29.519968 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.519972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.521028 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.521034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.522501 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.522510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.523796 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.523799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.523886 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.523908 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 20:14:29.523991 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 20:14:29.524057 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":220,"thread_start_us":138,"threads_started":1}
I20260926 20:14:29.528038 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 7be1f5485d374a8b9d6f45b25127b0e2 (DEFAULT_TABLE table=test-table [id=7ab9b43bb4ad43079c59a7cc90c7bb15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:29.527878 9298 tablet_service.cc:1467] Processing CreateTablet for tablet 7be1f5485d374a8b9d6f45b25127b0e2 (DEFAULT_TABLE table=test-table [id=7ab9b43bb4ad43079c59a7cc90c7bb15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:29.527880 9036 tablet_service.cc:1467] Processing CreateTablet for tablet 7be1f5485d374a8b9d6f45b25127b0e2 (DEFAULT_TABLE table=test-table [id=7ab9b43bb4ad43079c59a7cc90c7bb15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:29.529047 9298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be1f5485d374a8b9d6f45b25127b0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:29.529047 9036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be1f5485d374a8b9d6f45b25127b0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:29.529058 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be1f5485d374a8b9d6f45b25127b0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:29.535712 9381 tablet_bootstrap.cc:492] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9: Bootstrap starting.
I20260926 20:14:29.536958 9383 tablet_bootstrap.cc:492] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06: Bootstrap starting.
I20260926 20:14:29.537000 9382 tablet_bootstrap.cc:492] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6: Bootstrap starting.
I20260926 20:14:29.538511 9381 tablet_bootstrap.cc:654] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:29.538923 9382 tablet_bootstrap.cc:654] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:29.539053 9383 tablet_bootstrap.cc:654] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:29.539469 9381 log.cc:824] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:29.539803 9382 log.cc:824] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:29.539873 9383 log.cc:824] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:29.541774 9381 tablet_bootstrap.cc:492] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9: No bootstrap required, opened a new log
I20260926 20:14:29.541913 9382 tablet_bootstrap.cc:492] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6: No bootstrap required, opened a new log
I20260926 20:14:29.542057 9381 ts_tablet_manager.cc:1397] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 20:14:29.542102 9382 ts_tablet_manager.cc:1397] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:14:29.542202 9383 tablet_bootstrap.cc:492] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06: No bootstrap required, opened a new log
I20260926 20:14:29.542390 9383 ts_tablet_manager.cc:1397] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:14:29.545924 9381 raft_consensus.cc:348] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6cb48f61f584eacbb2a2d48418fdb06" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 39961 } } peers { permanent_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43237 } } peers { permanent_uuid: "6ee41549482449bd803064fee0403dd6" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45187 } }
I20260926 20:14:29.546191 9381 raft_consensus.cc:374] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:29.546244 9381 raft_consensus.cc:729] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087bce7ad6cb4cb6a29699b1a129b7a9, State: Initialized, Role: FOLLOWER
I20260926 20:14:29.546638 9383 raft_consensus.cc:348] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6cb48f61f584eacbb2a2d48418fdb06" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 39961 } } peers { permanent_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43237 } } peers { permanent_uuid: "6ee41549482449bd803064fee0403dd6" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45187 } }
I20260926 20:14:29.546638 9382 raft_consensus.cc:348] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6cb48f61f584eacbb2a2d48418fdb06" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 39961 } } peers { permanent_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43237 } } peers { permanent_uuid: "6ee41549482449bd803064fee0403dd6" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45187 } }
I20260926 20:14:29.546911 9382 raft_consensus.cc:374] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:29.546911 9383 raft_consensus.cc:374] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:29.546826 9381 consensus_queue.cc:260] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6cb48f61f584eacbb2a2d48418fdb06" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 39961 } } peers { permanent_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43237 } } peers { permanent_uuid: "6ee41549482449bd803064fee0403dd6" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45187 } }
I20260926 20:14:29.546962 9382 raft_consensus.cc:729] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee41549482449bd803064fee0403dd6, State: Initialized, Role: FOLLOWER
I20260926 20:14:29.546962 9383 raft_consensus.cc:729] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6cb48f61f584eacbb2a2d48418fdb06, State: Initialized, Role: FOLLOWER
I20260926 20:14:29.547446 9382 consensus_queue.cc:260] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6cb48f61f584eacbb2a2d48418fdb06" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 39961 } } peers { permanent_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43237 } } peers { permanent_uuid: "6ee41549482449bd803064fee0403dd6" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45187 } }
I20260926 20:14:29.547446 9383 consensus_queue.cc:260] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6cb48f61f584eacbb2a2d48418fdb06" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 39961 } } peers { permanent_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43237 } } peers { permanent_uuid: "6ee41549482449bd803064fee0403dd6" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45187 } }
I20260926 20:14:29.548192 9102 heartbeater.cc:503] Master 127.23.50.190:36283 was elected leader, sending a full tablet report...
I20260926 20:14:29.548522 9383 ts_tablet_manager.cc:1428] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:14:29.548522 9382 ts_tablet_manager.cc:1428] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:14:29.548878 9364 heartbeater.cc:503] Master 127.23.50.190:36283 was elected leader, sending a full tablet report...
I20260926 20:14:29.549199 9233 heartbeater.cc:503] Master 127.23.50.190:36283 was elected leader, sending a full tablet report...
I20260926 20:14:29.549787 9381 ts_tablet_manager.cc:1428] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
W20260926 20:14:29.567637 9103 tablet.cc:2369] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:14:29.586690 9388 raft_consensus.cc:482] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:14:29.586926 9388 raft_consensus.cc:504] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6cb48f61f584eacbb2a2d48418fdb06" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 39961 } } peers { permanent_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43237 } } peers { permanent_uuid: "6ee41549482449bd803064fee0403dd6" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45187 } }
I20260926 20:14:29.588135 9388 leader_election.cc:288] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6cb48f61f584eacbb2a2d48418fdb06 (127.23.50.131:39961), 087bce7ad6cb4cb6a29699b1a129b7a9 (127.23.50.130:43237)
I20260926 20:14:29.592159 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.588479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:52245 (local address 127.23.50.131:39961)
0926 20:14:29.588615 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:29.588620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:29.588850 (+ 230us) server_negotiation.cc:407] Connection header received
0926 20:14:29.588992 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:29.588996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:29.589063 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:29.589126 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:29.589571 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.590121 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.591181 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.591586 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.591681 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.591757 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:29.591949 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 20:14:29.592026 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:14:29.592176 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.588411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:43237 (local address 127.23.50.129:43785)
0926 20:14:29.588782 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:29.588821 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:29.588956 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:29.589243 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:29.589245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:29.589269 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:29.589415 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.589420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.590220 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.590223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.591091 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.591101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.591761 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.591766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.591852 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.591869 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 20:14:29.591963 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 20:14:29.592026 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":67,"thread_start_us":125,"threads_started":1}
I20260926 20:14:29.592170 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.588547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:43785 (local address 127.23.50.130:43237)
0926 20:14:29.588670 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:29.588675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:29.588928 (+ 253us) server_negotiation.cc:407] Connection header received
0926 20:14:29.589062 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:29.589064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:29.589133 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:29.589213 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:29.589530 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.590115 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.591237 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.591617 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.591754 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.591813 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:29.591996 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 20:14:29.592065 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:14:29.592185 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.588390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:39961 (local address 127.23.50.129:52245)
0926 20:14:29.588739 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:29.588750 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:29.588852 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:29.589171 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:29.589177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:29.589198 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:29.589421 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.589429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.590249 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.590252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.591065 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:29.591074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.591715 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.591719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:29.591811 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.591843 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 20:14:29.591918 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:14:29.592014 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":53,"thread_start_us":103,"threads_started":1}
I20260926 20:14:29.592832 9185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be1f5485d374a8b9d6f45b25127b0e2" candidate_uuid: "6ee41549482449bd803064fee0403dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9" is_pre_election: true
I20260926 20:14:29.592836 9318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be1f5485d374a8b9d6f45b25127b0e2" candidate_uuid: "6ee41549482449bd803064fee0403dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6cb48f61f584eacbb2a2d48418fdb06" is_pre_election: true
I20260926 20:14:29.593158 9318 raft_consensus.cc:2454] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ee41549482449bd803064fee0403dd6 in term 0.
I20260926 20:14:29.593158 9185 raft_consensus.cc:2454] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ee41549482449bd803064fee0403dd6 in term 0.
I20260926 20:14:29.593660 8989 leader_election.cc:302] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ee41549482449bd803064fee0403dd6, c6cb48f61f584eacbb2a2d48418fdb06; no voters:
I20260926 20:14:29.594022 9388 raft_consensus.cc:2781] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:14:29.594117 9388 raft_consensus.cc:482] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:14:29.594161 9388 raft_consensus.cc:3037] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:29.595460 9388 raft_consensus.cc:504] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6cb48f61f584eacbb2a2d48418fdb06" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 39961 } } peers { permanent_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43237 } } peers { permanent_uuid: "6ee41549482449bd803064fee0403dd6" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45187 } }
I20260926 20:14:29.595873 9388 leader_election.cc:288] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [CANDIDATE]: Term 1 election: Requested vote from peers c6cb48f61f584eacbb2a2d48418fdb06 (127.23.50.131:39961), 087bce7ad6cb4cb6a29699b1a129b7a9 (127.23.50.130:43237)
I20260926 20:14:29.596220 9318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be1f5485d374a8b9d6f45b25127b0e2" candidate_uuid: "6ee41549482449bd803064fee0403dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6cb48f61f584eacbb2a2d48418fdb06"
I20260926 20:14:29.596370 9185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be1f5485d374a8b9d6f45b25127b0e2" candidate_uuid: "6ee41549482449bd803064fee0403dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9"
I20260926 20:14:29.596380 9318 raft_consensus.cc:3037] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:29.596527 9185 raft_consensus.cc:3037] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:29.597630 9318 raft_consensus.cc:2454] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ee41549482449bd803064fee0403dd6 in term 1.
I20260926 20:14:29.597746 9185 raft_consensus.cc:2454] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ee41549482449bd803064fee0403dd6 in term 1.
I20260926 20:14:29.598014 8989 leader_election.cc:302] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ee41549482449bd803064fee0403dd6, c6cb48f61f584eacbb2a2d48418fdb06; no voters:
I20260926 20:14:29.598201 9388 raft_consensus.cc:2781] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:29.598436 9388 raft_consensus.cc:686] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [term 1 LEADER]: Becoming Leader. State: Replica: 6ee41549482449bd803064fee0403dd6, State: Running, Role: LEADER
I20260926 20:14:29.598773 9388 consensus_queue.cc:237] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6cb48f61f584eacbb2a2d48418fdb06" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 39961 } } peers { permanent_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43237 } } peers { permanent_uuid: "6ee41549482449bd803064fee0403dd6" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45187 } }
I20260926 20:14:29.602257 8914 catalog_manager.cc:5495] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ee41549482449bd803064fee0403dd6 (127.23.50.129). New cstate: current_term: 1 leader_uuid: "6ee41549482449bd803064fee0403dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6cb48f61f584eacbb2a2d48418fdb06" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 39961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ee41549482449bd803064fee0403dd6" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45187 } health_report { overall_health: HEALTHY } } }
I20260926 20:14:29.623183 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.619636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33258 (local address 127.23.50.129:45187)
0926 20:14:29.619786 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:29.619790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:29.619810 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:14:29.619884 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:29.619886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:29.619940 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:29.620054 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:14:29.620294 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.620814 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.621689 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:29.621973 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:29.622088 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:29.622403 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:14:29.622812 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:14:29.622950 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:29.622986 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 20:14:29.623041 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:14:29.629506 9185 raft_consensus.cc:1261] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9 [term 1 FOLLOWER]: Refusing update from remote peer 6ee41549482449bd803064fee0403dd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:14:29.629506 9318 raft_consensus.cc:1261] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06 [term 1 FOLLOWER]: Refusing update from remote peer 6ee41549482449bd803064fee0403dd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:14:29.630317 9392 consensus_queue.cc:1035] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6cb48f61f584eacbb2a2d48418fdb06" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 39961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:14:29.630548 9388 consensus_queue.cc:1035] T 7be1f5485d374a8b9d6f45b25127b0e2 P 6ee41549482449bd803064fee0403dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "087bce7ad6cb4cb6a29699b1a129b7a9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 43237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:14:29.640055 9396 mvcc.cc:205] Tried to move back new op lower bound from 7333698230793773056 to 7333698230679580672. Current Snapshot: MvccSnapshot[applied={T|T < 7333698230793773056 or (T in {7333698230793773056})}]
I20260926 20:14:29.646960 8913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:14:29.647158 8913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:14:29.649690 8913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33b86f086e5949639f3a43774b874e71: Sending DeleteTablet for 3 replicas of tablet 7be1f5485d374a8b9d6f45b25127b0e2
I20260926 20:14:29.650552 9036 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be1f5485d374a8b9d6f45b25127b0e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:29 UTC) from {username='slave'} at 127.0.0.1:33246
I20260926 20:14:29.650553 9298 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be1f5485d374a8b9d6f45b25127b0e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:29 UTC) from {username='slave'} at 127.0.0.1:35670
I20260926 20:14:29.650614 9167 tablet_service.cc:1514] Processing DeleteTablet for tablet 7be1f5485d374a8b9d6f45b25127b0e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:29 UTC) from {username='slave'} at 127.0.0.1:55218
I20260926 20:14:29.651240 9406 tablet_replica.cc:330] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06: stopping tablet replica
I20260926 20:14:29.651242 9407 tablet_replica.cc:330] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9: stopping tablet replica
I20260926 20:14:29.651588 9407 raft_consensus.cc:2229] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:14:29.651588 9406 raft_consensus.cc:2229] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:14:29.651924 9406 raft_consensus.cc:2258] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:29.651924 9407 raft_consensus.cc:2258] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:29.652057 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 8974
I20260926 20:14:29.654275 9406 ts_tablet_manager.cc:1905] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:14:29.655200 9407 ts_tablet_manager.cc:1905] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:14:29.657130 9406 ts_tablet_manager.cc:1918] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:14:29.657232 9406 log.cc:1196] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/7be1f5485d374a8b9d6f45b25127b0e2
I20260926 20:14:29.657555 9406 ts_tablet_manager.cc:1939] T 7be1f5485d374a8b9d6f45b25127b0e2 P c6cb48f61f584eacbb2a2d48418fdb06: Deleting consensus metadata
I20260926 20:14:29.658054 9407 ts_tablet_manager.cc:1918] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:14:29.658155 9407 log.cc:1196] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/7be1f5485d374a8b9d6f45b25127b0e2
I20260926 20:14:29.658458 9407 ts_tablet_manager.cc:1939] T 7be1f5485d374a8b9d6f45b25127b0e2 P 087bce7ad6cb4cb6a29699b1a129b7a9: Deleting consensus metadata
I20260926 20:14:29.658985 8898 catalog_manager.cc:4841] TS c6cb48f61f584eacbb2a2d48418fdb06 (127.23.50.131:39961): tablet 7be1f5485d374a8b9d6f45b25127b0e2 (table test-table [id=7ab9b43bb4ad43079c59a7cc90c7bb15]) successfully deleted
I20260926 20:14:29.659694 8900 catalog_manager.cc:4841] TS 087bce7ad6cb4cb6a29699b1a129b7a9 (127.23.50.130:43237): tablet 7be1f5485d374a8b9d6f45b25127b0e2 (table test-table [id=7ab9b43bb4ad43079c59a7cc90c7bb15]) successfully deleted
W20260926 20:14:29.659801 8899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:45187 (error 108)
I20260926 20:14:29.660568 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 9105
I20260926 20:14:29.660843 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:29.660530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:45187 (local address 127.0.0.1:33262)
0926 20:14:29.660658 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:29.660736 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:45187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:14:29.661677 8899 catalog_manager.cc:4584] TS 6ee41549482449bd803064fee0403dd6 (127.23.50.129:45187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7be1f5485d374a8b9d6f45b25127b0e2: Network error: Client connection negotiation failed: client connection to 127.23.50.129:45187: connect: Connection refused (error 111)
I20260926 20:14:29.666975 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 9236
I20260926 20:14:29.673481 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 8883
2026-09-26T20:14:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1062 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:29 dist-test-slave-7034 krb5kdc[9411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:29 dist-test-slave-7034 krb5kdc[9411](info): set up 2 sockets
Sep 26 20:14:29 dist-test-slave-7034 krb5kdc[9411](info): commencing operation
krb5kdc: starting...
W20260926 20:14:30.710340 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:30 dist-test-slave-7034 krb5kdc[9411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:14:30.836956 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:43657
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:43657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:30.953393 9424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:30.953751 9424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:30.953832 9424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:30.957901 9424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:30.957960 9424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:30.957979 9424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:30.957998 9424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:30.958012 9424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:30.963400 9424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:43657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:30.964447 9424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:30.965435 9424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:30.974159 9430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:30.974138 9429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:30.974130 9432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:30.974432 9424 server_base.cc:1034] running on GCE node
I20260926 20:14:30.975306 9424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:30.976537 9424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:30.977751 9424 hybrid_clock.cc:648] HybridClock initialized: now 1790453670977728 us; error 46 us; skew 500 ppm
Sep 26 20:14:30 dist-test-slave-7034 krb5kdc[9411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453670, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:30.980832 9424 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:14:30.981779 9424 webserver.cc:460] Webserver started at http://127.23.50.190:42719/ using document root <none> and password file <none>
I20260926 20:14:30.982404 9424 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:30.982470 9424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:30.982668 9424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:30.984839 9424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "6205a63a6df54622b518dbe9cb4163ad"
format_stamp: "Formatted at 2026-09-26 20:14:30 on dist-test-slave-7034"
server_key: "3b9678b980fedbf0afce88bd68f7ddf6"
server_key_iv: "1dc02db1c4410ccd99ea57d1815b1c55"
server_key_version: "encryptionkey@0"
I20260926 20:14:30.985394 9424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "6205a63a6df54622b518dbe9cb4163ad"
format_stamp: "Formatted at 2026-09-26 20:14:30 on dist-test-slave-7034"
server_key: "3b9678b980fedbf0afce88bd68f7ddf6"
server_key_iv: "1dc02db1c4410ccd99ea57d1815b1c55"
server_key_version: "encryptionkey@0"
I20260926 20:14:30.989401 9424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 20:14:30.992059 9439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:30.993392 9424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:14:30.993556 9424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "6205a63a6df54622b518dbe9cb4163ad"
format_stamp: "Formatted at 2026-09-26 20:14:30 on dist-test-slave-7034"
server_key: "3b9678b980fedbf0afce88bd68f7ddf6"
server_key_iv: "1dc02db1c4410ccd99ea57d1815b1c55"
server_key_version: "encryptionkey@0"
I20260926 20:14:30.993676 9424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:31.008987 9424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:31.012100 9424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:31.012326 9424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:31.019305 9424 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:43657
I20260926 20:14:31.019330 9491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:43657 every 8 connection(s)
I20260926 20:14:31.020236 9424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:31.023048 9492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:31.023453 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 9424
I20260926 20:14:31.023624 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:31.023910 23754 external_mini_cluster.cc:1428] Setting key 11bc5293aad4f1da85e4a29742ddf7dc
I20260926 20:14:31.028893 9492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad: Bootstrap starting.
I20260926 20:14:31.031230 9492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:31.031970 9492 log.cc:824] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:14:31 dist-test-slave-7034 krb5kdc[9411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:31.033983 9492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad: No bootstrap required, opened a new log
I20260926 20:14:31.037459 9492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6205a63a6df54622b518dbe9cb4163ad" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43657 } }
I20260926 20:14:31.037676 9492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:31.037735 9492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6205a63a6df54622b518dbe9cb4163ad, State: Initialized, Role: FOLLOWER
I20260926 20:14:31.038048 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.025384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.23.50.190:43657)
0926 20:14:31.025797 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:31.025814 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:31.025897 (+ 83us) server_negotiation.cc:407] Connection header received
0926 20:14:31.026570 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:31.026587 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:31.026909 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:31.027238 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:31.028063 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.029078 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.029990 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.030470 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.030649 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.033359 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:31.033371 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:31.033385 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:31.033411 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:31.035116 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:31.035575 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:31.035580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:31.035585 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:31.035694 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:31.035981 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:31.035985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:31.035986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:31.036374 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:31.036591 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:31.036900 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 20:14:31.037175 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":138,"threads_started":1}
I20260926 20:14:31.038273 9492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6205a63a6df54622b518dbe9cb4163ad" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43657 } }
I20260926 20:14:31.038437 9492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:31.038506 9492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:31.038607 9492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:31.039546 9492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6205a63a6df54622b518dbe9cb4163ad" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43657 } }
I20260926 20:14:31.039891 9492 leader_election.cc:302] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6205a63a6df54622b518dbe9cb4163ad; no voters:
I20260926 20:14:31.040298 9492 leader_election.cc:288] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:31.040396 9497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:31.040637 9497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [term 1 LEADER]: Becoming Leader. State: Replica: 6205a63a6df54622b518dbe9cb4163ad, State: Running, Role: LEADER
I20260926 20:14:31.040937 9497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6205a63a6df54622b518dbe9cb4163ad" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43657 } }
I20260926 20:14:31.041479 9492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:31.042750 9498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6205a63a6df54622b518dbe9cb4163ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6205a63a6df54622b518dbe9cb4163ad" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43657 } } }
I20260926 20:14:31.042761 9499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6205a63a6df54622b518dbe9cb4163ad. Latest consensus state: current_term: 1 leader_uuid: "6205a63a6df54622b518dbe9cb4163ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6205a63a6df54622b518dbe9cb4163ad" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43657 } } }
I20260926 20:14:31.042984 9498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:31.043018 9499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:31.043795 9509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:31.046515 9509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:31.052203 9509 catalog_manager.cc:1347] Generated new cluster ID: c3fc91ea526245c1874a067d0ff38d40
I20260926 20:14:31.052280 9509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:31.069968 9509 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:31.071918 9509 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:31.077925 9509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad: Generated new TSK 0
I20260926 20:14:31.078729 9509 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:14:31.108955 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43657
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:31.225832 9518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:31.226130 9518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:31.226176 9518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:31.230296 9518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:31.230370 9518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:31.230433 9518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:14:31.236055 9518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:31.237159 9518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:31.238234 9518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:31.248544 9523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:31.248524 9526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:31.248588 9518 server_base.cc:1034] running on GCE node
W20260926 20:14:31.248544 9524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:31.249261 9518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:31.250089 9518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:31.251471 9518 hybrid_clock.cc:648] HybridClock initialized: now 1790453671251449 us; error 36 us; skew 500 ppm
Sep 26 20:14:31 dist-test-slave-7034 krb5kdc[9411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453671, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:31.254751 9518 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:14:31.255678 9518 webserver.cc:460] Webserver started at http://127.23.50.129:46793/ using document root <none> and password file <none>
I20260926 20:14:31.256258 9518 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:31.256321 9518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:31.256495 9518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:31.258841 9518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "82219332641342ab9456fda8a61a87d0"
format_stamp: "Formatted at 2026-09-26 20:14:31 on dist-test-slave-7034"
server_key: "26ef165fdc699fd81f38053d0190af3c"
server_key_iv: "1b7069f5e7968390eb2d544aa70f6117"
server_key_version: "encryptionkey@0"
I20260926 20:14:31.259485 9518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "82219332641342ab9456fda8a61a87d0"
format_stamp: "Formatted at 2026-09-26 20:14:31 on dist-test-slave-7034"
server_key: "26ef165fdc699fd81f38053d0190af3c"
server_key_iv: "1b7069f5e7968390eb2d544aa70f6117"
server_key_version: "encryptionkey@0"
I20260926 20:14:31.263603 9518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:14:31.266340 9533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:31.267701 9518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:14:31.267859 9518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "82219332641342ab9456fda8a61a87d0"
format_stamp: "Formatted at 2026-09-26 20:14:31 on dist-test-slave-7034"
server_key: "26ef165fdc699fd81f38053d0190af3c"
server_key_iv: "1b7069f5e7968390eb2d544aa70f6117"
server_key_version: "encryptionkey@0"
I20260926 20:14:31.267990 9518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:31.290704 9518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:31.294206 9518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:31.294421 9518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:31.295090 9518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:31.296012 9518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:31.296083 9518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:31.296155 9518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:31.296198 9518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:31.304926 9518 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:36219
I20260926 20:14:31.305006 9646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:36219 every 8 connection(s)
I20260926 20:14:31.305910 9518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
Sep 26 20:14:31 dist-test-slave-7034 krb5kdc[9411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453671, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:31.315812 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 9518
I20260926 20:14:31.315963 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:14:31.316309 23754 external_mini_cluster.cc:1428] Setting key 0cc53c75f643b5f235122f172bba8516
I20260926 20:14:31.320284 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.307863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:33087 (local address 127.23.50.190:43657)
0926 20:14:31.308010 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:31.308014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:31.308856 (+ 842us) server_negotiation.cc:407] Connection header received
0926 20:14:31.310280 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:31.310283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:31.310342 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:31.310451 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:31.311676 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.312215 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.313410 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.313654 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.313772 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.316479 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:31.316482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:31.316484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:31.316510 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:31.318201 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:31.318732 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:31.318736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:31.318738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:31.318826 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:31.319186 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:31.319189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:31.319191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:31.319383 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:31.319488 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:31.320005 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 20:14:31.320128 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:14:31.321125 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.308263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43657 (local address 127.23.50.129:33087)
0926 20:14:31.308699 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:31.308742 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:31.310053 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:31.310630 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:31.310637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:31.311020 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:31.311512 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:31.311525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.312354 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.312364 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:31.313266 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:31.313277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.313817 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.313822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:31.314020 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.314585 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:14:31.314610 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:31.316293 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:31.318323 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:31.318329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:31.318346 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:31.318620 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:31.318927 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:31.318930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:31.318932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:31.319079 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:31.319503 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:31.319511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:31.319763 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 20:14:31.319966 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 20:14:31.320302 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":104,"threads_started":1}
I20260926 20:14:31.322618 9647 heartbeater.cc:347] Connected to a master server at 127.23.50.190:43657
I20260926 20:14:31.322866 9647 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:31.323374 9647 heartbeater.cc:511] Master 127.23.50.190:43657 requested a full tablet report, sending...
I20260926 20:14:31.324879 9456 ts_manager.cc:194] Registered new tserver with Master: 82219332641342ab9456fda8a61a87d0 (127.23.50.129:36219)
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
I20260926 20:14:31.326205 9456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:33087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:14:31.344786 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43657
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:31.457934 9652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:31.458359 9652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:31.458449 9652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:31.463603 9652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:31.463666 9652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:31.463728 9652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:14:31.469321 9652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:31.470563 9652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:31.471611 9652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:31.481470 9660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:31.481513 9658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:31.481513 9657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:31.482077 9652 server_base.cc:1034] running on GCE node
I20260926 20:14:31.482514 9652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:31.483175 9652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:31.484378 9652 hybrid_clock.cc:648] HybridClock initialized: now 1790453671484362 us; error 38 us; skew 500 ppm
Sep 26 20:14:31 dist-test-slave-7034 krb5kdc[9411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453671, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:31.487298 9652 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:14:31.488309 9652 webserver.cc:460] Webserver started at http://127.23.50.130:36827/ using document root <none> and password file <none>
I20260926 20:14:31.488933 9652 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:31.489029 9652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:31.489276 9652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:31.491469 9652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "b2a54b7e9dc84fdda38a706e79d6c2be"
format_stamp: "Formatted at 2026-09-26 20:14:31 on dist-test-slave-7034"
server_key: "5a7741bbe3d63b107ed730911becdab2"
server_key_iv: "1edc9ab6274d2ac97d691550faa0dee4"
server_key_version: "encryptionkey@0"
I20260926 20:14:31.492062 9652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "b2a54b7e9dc84fdda38a706e79d6c2be"
format_stamp: "Formatted at 2026-09-26 20:14:31 on dist-test-slave-7034"
server_key: "5a7741bbe3d63b107ed730911becdab2"
server_key_iv: "1edc9ab6274d2ac97d691550faa0dee4"
server_key_version: "encryptionkey@0"
I20260926 20:14:31.496438 9652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 20:14:31.499709 9667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:31.501180 9652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:14:31.501338 9652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "b2a54b7e9dc84fdda38a706e79d6c2be"
format_stamp: "Formatted at 2026-09-26 20:14:31 on dist-test-slave-7034"
server_key: "5a7741bbe3d63b107ed730911becdab2"
server_key_iv: "1edc9ab6274d2ac97d691550faa0dee4"
server_key_version: "encryptionkey@0"
I20260926 20:14:31.501479 9652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:31.516145 9652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:31.519498 9652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:31.519708 9652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:31.520334 9652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:31.521188 9652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:31.521260 9652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:31.521342 9652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:31.521394 9652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:31.530732 9652 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:33493
I20260926 20:14:31.530752 9780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:33493 every 8 connection(s)
I20260926 20:14:31.531678 9652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
Sep 26 20:14:31 dist-test-slave-7034 krb5kdc[9411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453671, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:31.541785 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 9652
I20260926 20:14:31.541977 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:14:31.542260 23754 external_mini_cluster.cc:1428] Setting key 705d6b91c9fc113a54fd1abb31c6f098
I20260926 20:14:31.546644 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.533477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:38049 (local address 127.23.50.190:43657)
0926 20:14:31.533628 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:31.533632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:31.534619 (+ 987us) server_negotiation.cc:407] Connection header received
0926 20:14:31.535999 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:31.536002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:31.536050 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:31.536127 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:31.537293 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.537988 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.539148 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.539382 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.539462 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.542551 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:31.542555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:31.542557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:31.542584 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:31.544316 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:31.544886 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:31.544893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:31.544898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:31.544994 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:31.545327 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:31.545333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:31.545338 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:31.545608 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:31.545727 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:31.546421 (+ 694us) server_negotiation.cc:299] Negotiation successful
0926 20:14:31.546526 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:14:31.547931 9783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.534010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43657 (local address 127.23.50.130:38049)
0926 20:14:31.534458 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:31.534503 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:31.535782 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:31.536289 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:31.536296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:31.536645 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:31.537137 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:31.537151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.538099 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.538102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:31.539038 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:31.539048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.539523 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.539526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:31.539679 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.540249 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:14:31.540266 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:31.542292 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:31.544465 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:31.544471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:31.544484 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:31.544759 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:31.545092 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:31.545095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:31.545096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:31.545236 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:31.545737 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:31.545747 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:31.546154 (+ 407us) client_negotiation.cc:769] Sending connection context
0926 20:14:31.546386 (+ 232us) client_negotiation.cc:240] Negotiation successful
0926 20:14:31.546780 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260926 20:14:31.549292 9781 heartbeater.cc:347] Connected to a master server at 127.23.50.190:43657
I20260926 20:14:31.549664 9781 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:31.550251 9781 heartbeater.cc:511] Master 127.23.50.190:43657 requested a full tablet report, sending...
I20260926 20:14:31.551411 9456 ts_manager.cc:194] Registered new tserver with Master: b2a54b7e9dc84fdda38a706e79d6c2be (127.23.50.130:33493)
I20260926 20:14:31.552026 9456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:38049
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:14:31.573557 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43657
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:31.686276 9786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:31.686553 9786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:31.686630 9786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:31.690771 9786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:31.690828 9786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:31.690945 9786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:14:31.696237 9786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:43657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:31.697414 9786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:31.698544 9786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:31.708128 9794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:31.708217 9791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:31.708217 9792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:31.708539 9786 server_base.cc:1034] running on GCE node
I20260926 20:14:31.708931 9786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:31.709628 9786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:31.710824 9786 hybrid_clock.cc:648] HybridClock initialized: now 1790453671710802 us; error 34 us; skew 500 ppm
Sep 26 20:14:31 dist-test-slave-7034 krb5kdc[9411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453671, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:31.713701 9786 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:14:31.714658 9786 webserver.cc:460] Webserver started at http://127.23.50.131:46479/ using document root <none> and password file <none>
I20260926 20:14:31.715238 9786 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:31.715341 9786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:31.715579 9786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:31.717795 9786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3"
format_stamp: "Formatted at 2026-09-26 20:14:31 on dist-test-slave-7034"
server_key: "4e4e789cf101a63ef55879036750bf3e"
server_key_iv: "2bfc94680c58546e25c0384bc68b4975"
server_key_version: "encryptionkey@0"
I20260926 20:14:31.718492 9786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3"
format_stamp: "Formatted at 2026-09-26 20:14:31 on dist-test-slave-7034"
server_key: "4e4e789cf101a63ef55879036750bf3e"
server_key_iv: "2bfc94680c58546e25c0384bc68b4975"
server_key_version: "encryptionkey@0"
I20260926 20:14:31.722922 9786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:14:31.725517 9801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:31.726994 9786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:14:31.727137 9786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3"
format_stamp: "Formatted at 2026-09-26 20:14:31 on dist-test-slave-7034"
server_key: "4e4e789cf101a63ef55879036750bf3e"
server_key_iv: "2bfc94680c58546e25c0384bc68b4975"
server_key_version: "encryptionkey@0"
I20260926 20:14:31.727263 9786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:31.746974 9786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:31.750679 9786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:31.750896 9786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:31.751533 9786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:31.752421 9786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:31.752507 9786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:31.752614 9786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:31.752668 9786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:31.762511 9786 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:42885
I20260926 20:14:31.762540 9914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:42885 every 8 connection(s)
I20260926 20:14:31.763511 9786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:14:31.770294 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 9786
I20260926 20:14:31.770437 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:14:31.770717 23754 external_mini_cluster.cc:1428] Setting key 646452b6db2b8c14df7253294d7a9514
Sep 26 20:14:31 dist-test-slave-7034 krb5kdc[9411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453671, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:31.777807 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.765460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:39559 (local address 127.23.50.190:43657)
0926 20:14:31.765595 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:31.765599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:31.766520 (+ 921us) server_negotiation.cc:407] Connection header received
0926 20:14:31.767885 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:31.767888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:31.767939 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:31.768020 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:31.769235 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.769790 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.771377 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.771703 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.771801 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.774623 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:31.774626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:31.774628 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:31.774654 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:31.776011 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:31.776500 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:31.776502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:31.776504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:31.776579 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:31.776893 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:31.776895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:31.776896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:31.777061 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:31.777135 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:31.777593 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 20:14:31.777688 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:14:31.778873 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.765930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43657 (local address 127.23.50.131:39559)
0926 20:14:31.766368 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:31.766411 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:31.767644 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:31.768201 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:31.768208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:31.768565 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:31.769075 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:31.769089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.770008 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.770013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:31.771229 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:31.771239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.771879 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.771883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:31.772085 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.772719 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:14:31.772740 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:31.774467 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:31.776125 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:31.776131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:31.776143 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:31.776409 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:31.776657 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:31.776659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:31.776660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:31.776792 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:31.777145 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:31.777150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:31.777390 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 20:14:31.777547 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 20:14:31.777811 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":104,"threads_started":1}
I20260926 20:14:31.780432 9915 heartbeater.cc:347] Connected to a master server at 127.23.50.190:43657
I20260926 20:14:31.780680 9915 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:31.781189 9915 heartbeater.cc:511] Master 127.23.50.190:43657 requested a full tablet report, sending...
I20260926 20:14:31.782572 9456 ts_manager.cc:194] Registered new tserver with Master: dc2c094a20cb4ecabfc4cf882b3a1fc3 (127.23.50.131:42885)
I20260926 20:14:31.783401 9456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:39559
I20260926 20:14:31.785056 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:14:31.794884 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.787398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.23.50.190:43657)
0926 20:14:31.787519 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:31.787524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:31.787713 (+ 189us) server_negotiation.cc:407] Connection header received
0926 20:14:31.787831 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:31.787834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:31.787873 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:31.787934 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:31.788646 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.789149 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.790052 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.790258 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.790346 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.791828 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:31.791831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:31.791833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:31.791856 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:31.793285 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:31.793762 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:31.793764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:31.793766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:31.793903 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:31.794165 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:31.794167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:31.794168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:31.794360 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:31.794474 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:31.794677 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 20:14:31.794751 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 20:14:31.797968 9456 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:14:31.801056 9456 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:14:31.816136 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.809921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.23.50.131:42885)
0926 20:14:31.811125 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:31.811139 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:31.811162 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:14:31.811244 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:31.811249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:31.811435 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:31.811600 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:31.811905 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.813057 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.814776 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.815383 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.815507 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.815668 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:31.815869 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 20:14:31.815978 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":166,"threads_started":1}
I20260926 20:14:31.816129 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.809720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:42885 (local address 127.0.0.1:60590)
0926 20:14:31.810580 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:31.810632 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:31.810766 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:31.811588 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:31.811594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:31.811616 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:31.811769 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:31.811777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.813189 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.813192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:31.814640 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:31.814651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.815514 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.815518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:31.815648 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.815693 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 20:14:31.815828 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 20:14:31.815978 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":122,"thread_start_us":130,"threads_started":1}
I20260926 20:14:31.816461 9930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.810577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.23.50.130:33493)
0926 20:14:31.811917 (+ 1340us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:31.811929 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:31.811952 (+ 23us) server_negotiation.cc:407] Connection header received
0926 20:14:31.812023 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:31.812027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:31.812198 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:31.812343 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:31.812562 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.813600 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.814842 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.815391 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.815523 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.815668 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:31.816206 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 20:14:31.816310 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1223,"thread_start_us":122,"threads_started":1}
I20260926 20:14:31.816138 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.809730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:33493 (local address 127.0.0.1:57670)
0926 20:14:31.810993 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:31.811008 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:31.811088 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:31.812343 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:31.812346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:31.812364 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:31.812441 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:31.812446 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.813742 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.813745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:31.814685 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:31.814695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.815540 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.815544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:31.815696 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.815720 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:14:31.815823 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:14:31.815968 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1195,"mutex_wait_us":65,"thread_start_us":128,"threads_started":1}
I20260926 20:14:31.817477 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.810053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36219 (local address 127.0.0.1:37560)
0926 20:14:31.811092 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:31.811108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:31.811198 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:31.811713 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:31.811716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:31.811733 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:31.811847 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:31.811855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.812963 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.812968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:31.814640 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:31.814651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.817094 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.817097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:31.817185 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.817209 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:14:31.817280 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 20:14:31.817360 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":163,"thread_start_us":93,"threads_started":1}
I20260926 20:14:31.818060 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:31.810602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.23.50.129:36219)
0926 20:14:31.811349 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:31.811361 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:31.811379 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:14:31.811439 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:31.811443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:31.811585 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:31.811739 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:31.811968 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.812839 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.815731 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:31.816779 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:31.816883 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:31.817028 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:31.817799 (+ 771us) server_negotiation.cc:299] Negotiation successful
0926 20:14:31.817926 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":100,"threads_started":1}
I20260926 20:14:31.819710 9849 tablet_service.cc:1467] Processing CreateTablet for tablet 745124e529eb4affaa87db21bbe10456 (DEFAULT_TABLE table=test-table [id=3847c7b87870400fa29eaffe49d01f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:31.819756 9715 tablet_service.cc:1467] Processing CreateTablet for tablet 745124e529eb4affaa87db21bbe10456 (DEFAULT_TABLE table=test-table [id=3847c7b87870400fa29eaffe49d01f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:31.820742 9849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745124e529eb4affaa87db21bbe10456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:31.820775 9715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745124e529eb4affaa87db21bbe10456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:31.820659 9581 tablet_service.cc:1467] Processing CreateTablet for tablet 745124e529eb4affaa87db21bbe10456 (DEFAULT_TABLE table=test-table [id=3847c7b87870400fa29eaffe49d01f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:31.821588 9581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745124e529eb4affaa87db21bbe10456. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:31.827227 9932 tablet_bootstrap.cc:492] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3: Bootstrap starting.
I20260926 20:14:31.828727 9933 tablet_bootstrap.cc:492] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be: Bootstrap starting.
I20260926 20:14:31.829551 9932 tablet_bootstrap.cc:654] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:31.830163 9934 tablet_bootstrap.cc:492] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0: Bootstrap starting.
I20260926 20:14:31.830686 9933 tablet_bootstrap.cc:654] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:31.830859 9932 log.cc:824] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:31.831969 9933 log.cc:824] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:31.832342 9934 tablet_bootstrap.cc:654] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:31.833144 9934 log.cc:824] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:31.833202 9932 tablet_bootstrap.cc:492] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3: No bootstrap required, opened a new log
I20260926 20:14:31.833422 9932 ts_tablet_manager.cc:1397] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:14:31.833897 9933 tablet_bootstrap.cc:492] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be: No bootstrap required, opened a new log
I20260926 20:14:31.834082 9933 ts_tablet_manager.cc:1397] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:14:31.835532 9934 tablet_bootstrap.cc:492] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0: No bootstrap required, opened a new log
I20260926 20:14:31.835716 9934 ts_tablet_manager.cc:1397] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:14:31.837033 9933 raft_consensus.cc:348] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42885 } } peers { permanent_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33493 } } peers { permanent_uuid: "82219332641342ab9456fda8a61a87d0" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36219 } }
I20260926 20:14:31.837280 9933 raft_consensus.cc:374] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:31.837342 9933 raft_consensus.cc:729] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a54b7e9dc84fdda38a706e79d6c2be, State: Initialized, Role: FOLLOWER
I20260926 20:14:31.837386 9932 raft_consensus.cc:348] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42885 } } peers { permanent_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33493 } } peers { permanent_uuid: "82219332641342ab9456fda8a61a87d0" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36219 } }
I20260926 20:14:31.837693 9932 raft_consensus.cc:374] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:31.837750 9932 raft_consensus.cc:729] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc2c094a20cb4ecabfc4cf882b3a1fc3, State: Initialized, Role: FOLLOWER
I20260926 20:14:31.837822 9933 consensus_queue.cc:260] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42885 } } peers { permanent_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33493 } } peers { permanent_uuid: "82219332641342ab9456fda8a61a87d0" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36219 } }
I20260926 20:14:31.838207 9932 consensus_queue.cc:260] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42885 } } peers { permanent_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33493 } } peers { permanent_uuid: "82219332641342ab9456fda8a61a87d0" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36219 } }
I20260926 20:14:31.838572 9781 heartbeater.cc:503] Master 127.23.50.190:43657 was elected leader, sending a full tablet report...
I20260926 20:14:31.838887 9933 ts_tablet_manager.cc:1428] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:14:31.839154 9932 ts_tablet_manager.cc:1428] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 20:14:31.839526 9915 heartbeater.cc:503] Master 127.23.50.190:43657 was elected leader, sending a full tablet report...
I20260926 20:14:31.840685 9934 raft_consensus.cc:348] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42885 } } peers { permanent_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33493 } } peers { permanent_uuid: "82219332641342ab9456fda8a61a87d0" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36219 } }
I20260926 20:14:31.840977 9934 raft_consensus.cc:374] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:31.841049 9934 raft_consensus.cc:729] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82219332641342ab9456fda8a61a87d0, State: Initialized, Role: FOLLOWER
I20260926 20:14:31.841501 9934 consensus_queue.cc:260] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42885 } } peers { permanent_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33493 } } peers { permanent_uuid: "82219332641342ab9456fda8a61a87d0" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36219 } }
I20260926 20:14:31.842238 9647 heartbeater.cc:503] Master 127.23.50.190:43657 was elected leader, sending a full tablet report...
I20260926 20:14:31.842510 9934 ts_tablet_manager.cc:1428] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 20:14:32.012805 9939 raft_consensus.cc:482] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:14:32.013078 9939 raft_consensus.cc:504] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42885 } } peers { permanent_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33493 } } peers { permanent_uuid: "82219332641342ab9456fda8a61a87d0" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36219 } }
I20260926 20:14:32.014437 9939 leader_election.cc:288] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2a54b7e9dc84fdda38a706e79d6c2be (127.23.50.130:33493), 82219332641342ab9456fda8a61a87d0 (127.23.50.129:36219)
W20260926 20:14:32.014700 9916 tablet.cc:2369] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:14:32.018693 9917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.014635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:33493 (local address 127.23.50.131:48305)
0926 20:14:32.014840 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:32.014858 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:32.014962 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:32.015378 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:32.015380 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:32.015393 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:32.015530 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:32.015537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.016461 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.016465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:32.017419 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:32.017428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.018228 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.018232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:32.018355 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:32.018386 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 20:14:32.018462 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 20:14:32.018530 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":26}
I20260926 20:14:32.018704 9930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.014796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:48305 (local address 127.23.50.130:33493)
0926 20:14:32.014943 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:32.014948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:32.014965 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:14:32.015101 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:32.015104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:32.015158 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:32.015233 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:32.015673 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.016336 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.017551 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.018101 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.018198 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:32.018240 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:32.018492 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 20:14:32.018582 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 20:14:32.018767 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.014705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36219 (local address 127.23.50.131:33841)
0926 20:14:32.015165 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:32.015178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:32.015251 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:32.015774 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:32.015777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:32.015801 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:32.015923 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:32.015930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.016748 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.016751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:32.017661 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:32.017671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.018361 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.018364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:32.018466 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:32.018491 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 20:14:32.018566 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:14:32.018631 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":40,"thread_start_us":129,"threads_started":1}
I20260926 20:14:32.018788 9931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.015385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:33841 (local address 127.23.50.129:36219)
0926 20:14:32.015539 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:32.015544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:32.015559 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:14:32.015616 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:32.015618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:32.015675 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:32.015733 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:32.016040 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.016637 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.017789 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.018249 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.018376 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:32.018430 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:32.018596 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 20:14:32.018664 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:14:32.019409 9601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745124e529eb4affaa87db21bbe10456" candidate_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82219332641342ab9456fda8a61a87d0" is_pre_election: true
I20260926 20:14:32.019407 9735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745124e529eb4affaa87db21bbe10456" candidate_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be" is_pre_election: true
I20260926 20:14:32.019717 9735 raft_consensus.cc:2454] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc2c094a20cb4ecabfc4cf882b3a1fc3 in term 0.
I20260926 20:14:32.019713 9601 raft_consensus.cc:2454] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc2c094a20cb4ecabfc4cf882b3a1fc3 in term 0.
I20260926 20:14:32.020222 9805 leader_election.cc:302] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82219332641342ab9456fda8a61a87d0, dc2c094a20cb4ecabfc4cf882b3a1fc3; no voters:
I20260926 20:14:32.020504 9939 raft_consensus.cc:2781] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:14:32.020594 9939 raft_consensus.cc:482] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:14:32.020633 9939 raft_consensus.cc:3037] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:32.021919 9939 raft_consensus.cc:504] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42885 } } peers { permanent_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33493 } } peers { permanent_uuid: "82219332641342ab9456fda8a61a87d0" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36219 } }
I20260926 20:14:32.022267 9939 leader_election.cc:288] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [CANDIDATE]: Term 1 election: Requested vote from peers b2a54b7e9dc84fdda38a706e79d6c2be (127.23.50.130:33493), 82219332641342ab9456fda8a61a87d0 (127.23.50.129:36219)
I20260926 20:14:32.022614 9601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745124e529eb4affaa87db21bbe10456" candidate_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82219332641342ab9456fda8a61a87d0"
I20260926 20:14:32.022629 9735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745124e529eb4affaa87db21bbe10456" candidate_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be"
I20260926 20:14:32.022734 9601 raft_consensus.cc:3037] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:32.022753 9735 raft_consensus.cc:3037] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:32.023886 9735 raft_consensus.cc:2454] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc2c094a20cb4ecabfc4cf882b3a1fc3 in term 1.
I20260926 20:14:32.023988 9601 raft_consensus.cc:2454] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc2c094a20cb4ecabfc4cf882b3a1fc3 in term 1.
I20260926 20:14:32.024220 9802 leader_election.cc:302] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2a54b7e9dc84fdda38a706e79d6c2be, dc2c094a20cb4ecabfc4cf882b3a1fc3; no voters:
I20260926 20:14:32.024410 9939 raft_consensus.cc:2781] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:32.024688 9939 raft_consensus.cc:686] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 1 LEADER]: Becoming Leader. State: Replica: dc2c094a20cb4ecabfc4cf882b3a1fc3, State: Running, Role: LEADER
I20260926 20:14:32.025022 9939 consensus_queue.cc:237] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42885 } } peers { permanent_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33493 } } peers { permanent_uuid: "82219332641342ab9456fda8a61a87d0" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36219 } }
I20260926 20:14:32.028223 9454 catalog_manager.cc:5495] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc2c094a20cb4ecabfc4cf882b3a1fc3 (127.23.50.131). New cstate: current_term: 1 leader_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc2c094a20cb4ecabfc4cf882b3a1fc3" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82219332641342ab9456fda8a61a87d0" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36219 } health_report { overall_health: UNKNOWN } } }
W20260926 20:14:32.032728 9782 tablet.cc:2369] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:14:32.052729 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.049079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.23.50.131:42885)
0926 20:14:32.049212 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:32.049217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:32.049233 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:14:32.049363 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:32.049366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:32.049419 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:32.049525 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:14:32.049785 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.050363 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.051223 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.051451 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.051561 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:32.051859 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:14:32.052267 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:14:32.052441 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:32.052531 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 20:14:32.052571 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 20:14:32.057346 9648 tablet.cc:2369] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:14:32.058471 9601 raft_consensus.cc:1261] T 745124e529eb4affaa87db21bbe10456 P 82219332641342ab9456fda8a61a87d0 [term 1 FOLLOWER]: Refusing update from remote peer dc2c094a20cb4ecabfc4cf882b3a1fc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:14:32.058472 9735 raft_consensus.cc:1261] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be [term 1 FOLLOWER]: Refusing update from remote peer dc2c094a20cb4ecabfc4cf882b3a1fc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:14:32.059105 9939 consensus_queue.cc:1035] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2a54b7e9dc84fdda38a706e79d6c2be" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 33493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:14:32.059300 9942 consensus_queue.cc:1035] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82219332641342ab9456fda8a61a87d0" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:14:32.067919 9946 mvcc.cc:205] Tried to move back new op lower bound from 7333698240744321024 to 7333698240616939520. Current Snapshot: MvccSnapshot[applied={T|T < 7333698240744321024}]
I20260926 20:14:32.076851 9455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:14:32.077009 9455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:14:32.079567 9455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6205a63a6df54622b518dbe9cb4163ad: Sending DeleteTablet for 3 replicas of tablet 745124e529eb4affaa87db21bbe10456
I20260926 20:14:32.080376 9849 tablet_service.cc:1514] Processing DeleteTablet for tablet 745124e529eb4affaa87db21bbe10456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:32 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:60590
I20260926 20:14:32.080519 9581 tablet_service.cc:1514] Processing DeleteTablet for tablet 745124e529eb4affaa87db21bbe10456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:32 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:37560
I20260926 20:14:32.080904 9955 tablet_replica.cc:330] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3: stopping tablet replica
I20260926 20:14:32.080915 9715 tablet_service.cc:1514] Processing DeleteTablet for tablet 745124e529eb4affaa87db21bbe10456 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:32 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:57670
I20260926 20:14:32.081238 9955 raft_consensus.cc:2229] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:14:32.081434 9957 tablet_replica.cc:330] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be: stopping tablet replica
I20260926 20:14:32.081662 9955 raft_consensus.cc:2258] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:32.081764 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 9518
I20260926 20:14:32.082517 9957 raft_consensus.cc:2229] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:14:32.082888 9957 raft_consensus.cc:2258] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:32.084364 9957 ts_tablet_manager.cc:1905] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:14:32.084522 9955 ts_tablet_manager.cc:1905] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:14:32.087324 9957 ts_tablet_manager.cc:1918] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:14:32.087510 9957 log.cc:1196] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/745124e529eb4affaa87db21bbe10456
I20260926 20:14:32.087893 9957 ts_tablet_manager.cc:1939] T 745124e529eb4affaa87db21bbe10456 P b2a54b7e9dc84fdda38a706e79d6c2be: Deleting consensus metadata
I20260926 20:14:32.088133 9955 ts_tablet_manager.cc:1918] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:14:32.088240 9955 log.cc:1196] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/745124e529eb4affaa87db21bbe10456
I20260926 20:14:32.088579 9955 ts_tablet_manager.cc:1939] T 745124e529eb4affaa87db21bbe10456 P dc2c094a20cb4ecabfc4cf882b3a1fc3: Deleting consensus metadata
I20260926 20:14:32.089697 9441 catalog_manager.cc:4841] TS dc2c094a20cb4ecabfc4cf882b3a1fc3 (127.23.50.131:42885): tablet 745124e529eb4affaa87db21bbe10456 (table test-table [id=3847c7b87870400fa29eaffe49d01f70]) successfully deleted
I20260926 20:14:32.089697 9440 catalog_manager.cc:4841] TS b2a54b7e9dc84fdda38a706e79d6c2be (127.23.50.130:33493): tablet 745124e529eb4affaa87db21bbe10456 (table test-table [id=3847c7b87870400fa29eaffe49d01f70]) successfully deleted
W20260926 20:14:32.090301 9443 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:36219 (error 108)
I20260926 20:14:32.091308 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.090987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:36219 (local address 127.0.0.1:37570)
0926 20:14:32.091134 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:32.091213 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:36219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 20:14:32.091382 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 9652
W20260926 20:14:32.092108 9443 catalog_manager.cc:4584] TS 82219332641342ab9456fda8a61a87d0 (127.23.50.129:36219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 745124e529eb4affaa87db21bbe10456: Network error: Client connection negotiation failed: client connection to 127.23.50.129:36219: connect: Connection refused (error 111)
I20260926 20:14:32.097072 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 9786
I20260926 20:14:32.103195 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 9424
2026-09-26T20:14:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2429 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T20:14:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:32Z Disabled control of system clock
I20260926 20:14:32.133939 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:46707
--webserver_interface=127.23.50.190
--webserver_port=0
--builtin_ntp_servers=127.23.50.148:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:46707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:14:32.248569 9962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:32.248924 9962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:32.248988 9962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:32.253150 9962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:32.253221 9962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:32.253237 9962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:32.253255 9962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:32.253270 9962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:32.258496 9962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:46707
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:46707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:32.259721 9962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:32.261078 9962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:32.269666 9968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:32.269666 9967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:32.269757 9970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:32.269752 9962 server_base.cc:1034] running on GCE node
I20260926 20:14:32.270671 9962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:32.271752 9962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:32.272949 9962 hybrid_clock.cc:648] HybridClock initialized: now 1790453672272928 us; error 33 us; skew 500 ppm
I20260926 20:14:32.274887 9962 webserver.cc:460] Webserver started at http://127.23.50.190:46351/ using document root <none> and password file <none>
I20260926 20:14:32.275477 9962 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:32.275571 9962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:32.275815 9962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:32.277828 9962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "709c911ff5eb4ca8af7cb34485e964fb"
format_stamp: "Formatted at 2026-09-26 20:14:32 on dist-test-slave-7034"
server_key: "f4c03bf2f0a3dedff07c7ec430513e92"
server_key_iv: "798b1021d64ed73c929d3affd71aa53d"
server_key_version: "encryptionkey@0"
I20260926 20:14:32.278462 9962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "709c911ff5eb4ca8af7cb34485e964fb"
format_stamp: "Formatted at 2026-09-26 20:14:32 on dist-test-slave-7034"
server_key: "f4c03bf2f0a3dedff07c7ec430513e92"
server_key_iv: "798b1021d64ed73c929d3affd71aa53d"
server_key_version: "encryptionkey@0"
I20260926 20:14:32.282145 9962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260926 20:14:32.284559 9976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:32.285987 9962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:32.286130 9962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "709c911ff5eb4ca8af7cb34485e964fb"
format_stamp: "Formatted at 2026-09-26 20:14:32 on dist-test-slave-7034"
server_key: "f4c03bf2f0a3dedff07c7ec430513e92"
server_key_iv: "798b1021d64ed73c929d3affd71aa53d"
server_key_version: "encryptionkey@0"
I20260926 20:14:32.286247 9962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:32.316357 9962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:32.317094 9962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:32.317301 9962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:32.324024 9962 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:46707
I20260926 20:14:32.324033 10028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:46707 every 8 connection(s)
I20260926 20:14:32.325009 9962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:32.328161 10029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:32.330403 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 9962
I20260926 20:14:32.330531 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:32.330862 23754 external_mini_cluster.cc:1428] Setting key deea11d8da89f4f5da5654ee1a7b14b8
I20260926 20:14:32.334488 10029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb: Bootstrap starting.
I20260926 20:14:32.336670 10029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:32.337409 10029 log.cc:824] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:32.339416 10029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb: No bootstrap required, opened a new log
I20260926 20:14:32.340094 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.332273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.23.50.190:46707)
0926 20:14:32.332886 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:32.332901 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:32.332934 (+ 33us) server_negotiation.cc:407] Connection header received
0926 20:14:32.333648 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:32.333666 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:32.334120 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:32.334452 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:32.334868 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.335807 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.336706 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.337247 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.337430 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:32.338313 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:32.338321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:32.338338 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:32.338722 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:32.338778 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:32.338797 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:32.338923 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:32.339072 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 20:14:32.339336 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260926 20:14:32.343466 10029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709c911ff5eb4ca8af7cb34485e964fb" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46707 } }
I20260926 20:14:32.343724 10029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:32.343814 10029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709c911ff5eb4ca8af7cb34485e964fb, State: Initialized, Role: FOLLOWER
I20260926 20:14:32.344238 10029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709c911ff5eb4ca8af7cb34485e964fb" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46707 } }
I20260926 20:14:32.344394 10029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:32.344478 10029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:32.344600 10029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:32.345556 10029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709c911ff5eb4ca8af7cb34485e964fb" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46707 } }
I20260926 20:14:32.345984 10029 leader_election.cc:302] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 709c911ff5eb4ca8af7cb34485e964fb; no voters:
I20260926 20:14:32.346312 10029 leader_election.cc:288] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:32.346417 10034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:32.346661 10034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [term 1 LEADER]: Becoming Leader. State: Replica: 709c911ff5eb4ca8af7cb34485e964fb, State: Running, Role: LEADER
I20260926 20:14:32.346974 10034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709c911ff5eb4ca8af7cb34485e964fb" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46707 } }
I20260926 20:14:32.347671 10029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:32.348775 10036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 709c911ff5eb4ca8af7cb34485e964fb. Latest consensus state: current_term: 1 leader_uuid: "709c911ff5eb4ca8af7cb34485e964fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709c911ff5eb4ca8af7cb34485e964fb" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46707 } } }
I20260926 20:14:32.348928 10036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:32.349195 10035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "709c911ff5eb4ca8af7cb34485e964fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "709c911ff5eb4ca8af7cb34485e964fb" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 46707 } } }
I20260926 20:14:32.349298 10035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:32.349387 10043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:32.352676 10043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:32.358808 10043 catalog_manager.cc:1347] Generated new cluster ID: d9ec1b5ea7c84509b97623aac165569e
I20260926 20:14:32.358894 10043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:32.384063 10043 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:32.385052 10043 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:32.393421 10043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb: Generated new TSK 0
I20260926 20:14:32.394287 10043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:32.407447 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:46707
--builtin_ntp_servers=127.23.50.148:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:14:32.521528 10053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:32.521811 10053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:32.521926 10053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:32.526288 10053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:32.526362 10053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:32.526422 10053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:14:32.531786 10053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:46707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:32.533272 10053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:32.534904 10053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:32.544984 10059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:32.545020 10061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:32.545183 10053 server_base.cc:1034] running on GCE node
W20260926 20:14:32.545030 10058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:32.545814 10053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:32.546521 10053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:32.547709 10053 hybrid_clock.cc:648] HybridClock initialized: now 1790453672547682 us; error 31 us; skew 500 ppm
I20260926 20:14:32.550012 10053 webserver.cc:460] Webserver started at http://127.23.50.129:35541/ using document root <none> and password file <none>
I20260926 20:14:32.550621 10053 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:32.550686 10053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:32.550920 10053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:32.553158 10053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "0db2af88f2684b10ae72412facb8af38"
format_stamp: "Formatted at 2026-09-26 20:14:32 on dist-test-slave-7034"
server_key: "d6d84b465004fcc9be8dc2429686c513"
server_key_iv: "fe8527ea644f5344e508347b48f2c0e5"
server_key_version: "encryptionkey@0"
I20260926 20:14:32.553917 10053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "0db2af88f2684b10ae72412facb8af38"
format_stamp: "Formatted at 2026-09-26 20:14:32 on dist-test-slave-7034"
server_key: "d6d84b465004fcc9be8dc2429686c513"
server_key_iv: "fe8527ea644f5344e508347b48f2c0e5"
server_key_version: "encryptionkey@0"
I20260926 20:14:32.558207 10053 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:14:32.560681 10067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:32.562086 10053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:32.562230 10053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "0db2af88f2684b10ae72412facb8af38"
format_stamp: "Formatted at 2026-09-26 20:14:32 on dist-test-slave-7034"
server_key: "d6d84b465004fcc9be8dc2429686c513"
server_key_iv: "fe8527ea644f5344e508347b48f2c0e5"
server_key_version: "encryptionkey@0"
I20260926 20:14:32.562356 10053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:32.593279 10053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:32.594105 10053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:32.594313 10053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:32.594944 10053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:32.595959 10053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:32.596033 10053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:32.596103 10053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:32.596158 10053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:32.605489 10053 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:44877
I20260926 20:14:32.605513 10180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:44877 every 8 connection(s)
I20260926 20:14:32.606454 10053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:14:32.614466 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 10053
I20260926 20:14:32.614640 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:14:32.615000 23754 external_mini_cluster.cc:1428] Setting key fcf2616c7a2ed6e394a7e868bcacef39
I20260926 20:14:32.616885 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.608484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:53197 (local address 127.23.50.190:46707)
0926 20:14:32.608638 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:32.608643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:32.609925 (+ 1282us) server_negotiation.cc:407] Connection header received
0926 20:14:32.611395 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:32.611397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:32.611446 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:32.611527 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:32.612447 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.612971 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.614416 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.614727 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.614823 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:32.616273 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:32.616277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:32.616279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:32.616291 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:32.616310 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:32.616317 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:32.616406 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:32.616718 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 20:14:32.616772 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 20:14:32.617278 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:46707
--builtin_ntp_servers=127.23.50.148:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:14:32.618049 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.608950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:46707 (local address 127.23.50.129:53197)
0926 20:14:32.609677 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:32.609725 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:32.611188 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:32.611684 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:32.611690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:32.611779 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:32.612302 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:32.612315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.613110 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.613114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:32.614281 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:32.614296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.614894 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.614899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:32.615127 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:32.615973 (+ 846us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:14:32.615975 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:32.616017 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:14:32.616080 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:32.616527 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:32.616535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:32.616539 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:14:32.616687 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 20:14:32.616983 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":112,"threads_started":1}
I20260926 20:14:32.619272 10181 heartbeater.cc:347] Connected to a master server at 127.23.50.190:46707
I20260926 20:14:32.619529 10181 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:32.620040 10181 heartbeater.cc:511] Master 127.23.50.190:46707 requested a full tablet report, sending...
I20260926 20:14:32.621716 9993 ts_manager.cc:194] Registered new tserver with Master: 0db2af88f2684b10ae72412facb8af38 (127.23.50.129:44877)
I20260926 20:14:32.623497 9993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.50.129:53197
W20260926 20:14:32.734933 10184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:32.735241 10184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:32.735283 10184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:32.739472 10184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:32.739539 10184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:32.739596 10184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:14:32.744503 10184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:46707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:32.745565 10184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:32.746978 10184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:14:32.757422 10184 server_base.cc:1034] running on GCE node
W20260926 20:14:32.757359 10189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:32.757339 10192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:32.757370 10190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:32.758162 10184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:32.758869 10184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:32.760080 10184 hybrid_clock.cc:648] HybridClock initialized: now 1790453672760062 us; error 31 us; skew 500 ppm
I20260926 20:14:32.762138 10184 webserver.cc:460] Webserver started at http://127.23.50.130:41133/ using document root <none> and password file <none>
I20260926 20:14:32.762742 10184 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:32.762859 10184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:32.763090 10184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:32.765413 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "65c27f9d635c47d9a14c3c095c3c03d9"
format_stamp: "Formatted at 2026-09-26 20:14:32 on dist-test-slave-7034"
server_key: "51335219864fb0e8a713082ed7ef41ac"
server_key_iv: "87956789a97e9de565f6edc579f6f8cd"
server_key_version: "encryptionkey@0"
I20260926 20:14:32.766135 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "65c27f9d635c47d9a14c3c095c3c03d9"
format_stamp: "Formatted at 2026-09-26 20:14:32 on dist-test-slave-7034"
server_key: "51335219864fb0e8a713082ed7ef41ac"
server_key_iv: "87956789a97e9de565f6edc579f6f8cd"
server_key_version: "encryptionkey@0"
I20260926 20:14:32.770112 10184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:14:32.772503 10198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:32.773808 10184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:14:32.773967 10184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "65c27f9d635c47d9a14c3c095c3c03d9"
format_stamp: "Formatted at 2026-09-26 20:14:32 on dist-test-slave-7034"
server_key: "51335219864fb0e8a713082ed7ef41ac"
server_key_iv: "87956789a97e9de565f6edc579f6f8cd"
server_key_version: "encryptionkey@0"
I20260926 20:14:32.774084 10184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:32.794329 10184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:32.795040 10184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:32.795248 10184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:32.795849 10184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:32.796818 10184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:32.796864 10184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:32.796903 10184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:32.796955 10184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:32.805584 10184 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:41923
I20260926 20:14:32.805608 10311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:41923 every 8 connection(s)
I20260926 20:14:32.806581 10184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:14:32.814379 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 10184
I20260926 20:14:32.814559 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:14:32.814904 23754 external_mini_cluster.cc:1428] Setting key 7b197833ac659ac28d392204fdc56b86
I20260926 20:14:32.816347 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.808551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:50395 (local address 127.23.50.190:46707)
0926 20:14:32.808732 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:32.808737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:32.809940 (+ 1203us) server_negotiation.cc:407] Connection header received
0926 20:14:32.811317 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:32.811320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:32.811367 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:32.811445 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:32.812353 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.812886 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.814048 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.814409 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.814533 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:32.815838 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:32.815841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:32.815845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:32.815857 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:32.815882 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:32.815891 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:32.815969 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:32.816115 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 20:14:32.816180 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 20:14:32.817073 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:46707
--builtin_ntp_servers=127.23.50.148:44133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:14:32.817338 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.809071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:46707 (local address 127.23.50.130:50395)
0926 20:14:32.809741 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:32.809782 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:32.811127 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:32.811615 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:32.811622 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:32.811707 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:32.812188 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:32.812200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.813000 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.813003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:32.813918 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:32.813929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:32.814607 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:32.814612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:32.814791 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:32.815519 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:14:32.815521 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:32.815577 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:14:32.815662 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:32.815978 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:32.815985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:32.815990 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 20:14:32.816085 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 20:14:32.816394 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":113,"threads_started":1}
I20260926 20:14:32.818600 10312 heartbeater.cc:347] Connected to a master server at 127.23.50.190:46707
I20260926 20:14:32.818857 10312 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:32.819489 10312 heartbeater.cc:511] Master 127.23.50.190:46707 requested a full tablet report, sending...
I20260926 20:14:32.820729 9993 ts_manager.cc:194] Registered new tserver with Master: 65c27f9d635c47d9a14c3c095c3c03d9 (127.23.50.130:41923)
I20260926 20:14:32.821400 9993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.50.130:50395
W20260926 20:14:32.933175 10315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:32.933491 10315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:32.933560 10315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:32.937585 10315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:32.937678 10315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:32.937773 10315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:14:32.942576 10315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:44133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:46707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:32.943725 10315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:32.945155 10315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:32.954828 10321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:32.954828 10320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:32.954933 10315 server_base.cc:1034] running on GCE node
W20260926 20:14:32.955081 10323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:32.955492 10315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:32.956072 10315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:32.957234 10315 hybrid_clock.cc:648] HybridClock initialized: now 1790453672957223 us; error 26 us; skew 500 ppm
I20260926 20:14:32.959095 10315 webserver.cc:460] Webserver started at http://127.23.50.131:33321/ using document root <none> and password file <none>
I20260926 20:14:32.959623 10315 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:32.959684 10315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:32.959847 10315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:32.961813 10315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "0ae334066d1247e3beda763b40e715ee"
format_stamp: "Formatted at 2026-09-26 20:14:32 on dist-test-slave-7034"
server_key: "c2853300f55d809dfb26fdc1ca42b37b"
server_key_iv: "b85c48311ab60bd0c5df91d5e9309bb6"
server_key_version: "encryptionkey@0"
I20260926 20:14:32.962464 10315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "0ae334066d1247e3beda763b40e715ee"
format_stamp: "Formatted at 2026-09-26 20:14:32 on dist-test-slave-7034"
server_key: "c2853300f55d809dfb26fdc1ca42b37b"
server_key_iv: "b85c48311ab60bd0c5df91d5e9309bb6"
server_key_version: "encryptionkey@0"
I20260926 20:14:32.966550 10315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:14:32.969141 10329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:32.970465 10315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:32.970562 10315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "0ae334066d1247e3beda763b40e715ee"
format_stamp: "Formatted at 2026-09-26 20:14:32 on dist-test-slave-7034"
server_key: "c2853300f55d809dfb26fdc1ca42b37b"
server_key_iv: "b85c48311ab60bd0c5df91d5e9309bb6"
server_key_version: "encryptionkey@0"
I20260926 20:14:32.970652 10315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:32.984186 10315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:32.984813 10315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:32.984964 10315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:32.985481 10315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:32.986481 10315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:32.986529 10315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:32.986567 10315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:32.986582 10315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:32.996002 10315 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:34931
I20260926 20:14:32.996027 10442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:34931 every 8 connection(s)
I20260926 20:14:32.996927 10315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:14:33.004040 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 10315
I20260926 20:14:33.004148 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:14:33.004421 23754 external_mini_cluster.cc:1428] Setting key e8af192adf77aab7d10cd7ebe0689951
I20260926 20:14:33.006325 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.998856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:48581 (local address 127.23.50.190:46707)
0926 20:14:32.999025 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:32.999030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:33.000121 (+ 1091us) server_negotiation.cc:407] Connection header received
0926 20:14:33.001475 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:33.001477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:33.001523 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:33.001598 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:33.002552 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.003083 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.004292 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.004564 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.004691 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.005783 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:33.005787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:33.005789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:33.005809 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:33.005829 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:33.005837 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:33.005943 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:33.006108 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 20:14:33.006174 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:14:33.007364 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:32.999260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:46707 (local address 127.23.50.131:48581)
0926 20:14:32.999963 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:33.000005 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:33.001276 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:33.001765 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:33.001772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:33.001925 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:33.002422 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:33.002436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.003198 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.003201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:33.004081 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:33.004090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.004736 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.004740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:33.004927 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.005521 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:14:33.005523 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:33.005562 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:14:33.005628 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:33.005944 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:33.005956 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:33.005963 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 20:14:33.006096 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 20:14:33.006380 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":113,"threads_started":1}
I20260926 20:14:33.008388 10443 heartbeater.cc:347] Connected to a master server at 127.23.50.190:46707
I20260926 20:14:33.008623 10443 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:33.009138 10443 heartbeater.cc:511] Master 127.23.50.190:46707 requested a full tablet report, sending...
I20260926 20:14:33.010123 9993 ts_manager.cc:194] Registered new tserver with Master: 0ae334066d1247e3beda763b40e715ee (127.23.50.131:34931)
I20260926 20:14:33.010694 9993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.50.131:48581
I20260926 20:14:33.018815 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:14:33.024353 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.020735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.23.50.190:46707)
0926 20:14:33.020921 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:33.020925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:33.020997 (+ 72us) server_negotiation.cc:407] Connection header received
0926 20:14:33.021100 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:33.021103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:33.021142 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:33.021209 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:33.021469 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.022001 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.022763 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.022974 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.023048 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.023870 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:33.023873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:33.023875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:33.023885 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:33.023902 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:33.023909 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:33.024008 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:33.024129 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 20:14:33.024184 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 20:14:33.027159 9993 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:14:33.030207 9993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:14:33.044116 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.038149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:41923 (local address 127.0.0.1:58600)
0926 20:14:33.038953 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:33.038967 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:33.039064 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:33.040027 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:33.040030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:33.040050 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:33.040140 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:33.040147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.041486 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.041489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:33.042475 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:33.042485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.043558 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.043561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:33.043669 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.043707 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 20:14:33.043833 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 20:14:33.043949 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":86,"spinlock_wait_cycles":6912,"thread_start_us":95,"threads_started":1}
I20260926 20:14:33.044157 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.038871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.23.50.130:41923)
0926 20:14:33.039529 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:33.039544 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:33.039570 (+ 26us) server_negotiation.cc:407] Connection header received
0926 20:14:33.039647 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:33.039652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:33.039861 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:33.040051 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:33.040254 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.041351 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.042606 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.043435 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.043552 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.043759 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:33.043874 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 20:14:33.043974 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":148,"threads_started":1}
I20260926 20:14:33.044590 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.038419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:34931 (local address 127.0.0.1:42254)
0926 20:14:33.039269 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:33.039281 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:33.039348 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:33.040198 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:33.040202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:33.040224 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:33.040296 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:33.040301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.041257 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.041261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:33.042691 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:33.042703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.043866 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.043869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:33.043954 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.043978 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 20:14:33.044048 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 20:14:33.044117 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":105,"thread_start_us":84,"threads_started":1}
I20260926 20:14:33.044989 10457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.038071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:44877 (local address 127.0.0.1:52996)
0926 20:14:33.038877 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:33.038926 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:33.039064 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:33.039840 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:33.039846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:33.039867 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:33.039998 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:33.040009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.041443 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.041446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:33.043255 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:33.043264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.044591 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.044594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:33.044694 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.044721 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:14:33.044781 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 20:14:33.044856 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":75,"spinlock_wait_cycles":384,"thread_start_us":81,"threads_started":1}
I20260926 20:14:33.045032 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.038883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.23.50.131:34931)
0926 20:14:33.039599 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:33.039613 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:33.039638 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:14:33.039706 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:33.039711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:33.039909 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:33.040066 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:33.040420 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.041136 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.042800 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.043566 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.043657 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.043799 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:33.044194 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 20:14:33.044879 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":143,"threads_started":1}
I20260926 20:14:33.045542 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.038185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.23.50.129:44877)
0926 20:14:33.038577 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:33.038588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:33.039208 (+ 620us) server_negotiation.cc:407] Connection header received
0926 20:14:33.039309 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:33.039317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:33.039535 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:33.039683 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:33.040128 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.041330 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.043386 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.044167 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.044595 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.044951 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:33.045001 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 20:14:33.045368 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":86,"threads_started":1}
I20260926 20:14:33.047426 10246 tablet_service.cc:1467] Processing CreateTablet for tablet 02d3851701784e1da0bb2f5ef78c3d50 (DEFAULT_TABLE table=test-table [id=16a6b444f3f947beb5c255125d27103a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:33.047749 10377 tablet_service.cc:1467] Processing CreateTablet for tablet 02d3851701784e1da0bb2f5ef78c3d50 (DEFAULT_TABLE table=test-table [id=16a6b444f3f947beb5c255125d27103a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:33.047806 10115 tablet_service.cc:1467] Processing CreateTablet for tablet 02d3851701784e1da0bb2f5ef78c3d50 (DEFAULT_TABLE table=test-table [id=16a6b444f3f947beb5c255125d27103a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:33.048619 10246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3851701784e1da0bb2f5ef78c3d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:33.048687 10377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3851701784e1da0bb2f5ef78c3d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:33.048826 10115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02d3851701784e1da0bb2f5ef78c3d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:33.056999 10462 tablet_bootstrap.cc:492] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee: Bootstrap starting.
I20260926 20:14:33.059429 10460 tablet_bootstrap.cc:492] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9: Bootstrap starting.
I20260926 20:14:33.059572 10462 tablet_bootstrap.cc:654] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:33.060438 10462 log.cc:824] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:33.061219 10461 tablet_bootstrap.cc:492] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38: Bootstrap starting.
I20260926 20:14:33.061754 10460 tablet_bootstrap.cc:654] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:33.062707 10460 log.cc:824] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:33.064280 10462 tablet_bootstrap.cc:492] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee: No bootstrap required, opened a new log
I20260926 20:14:33.064502 10462 ts_tablet_manager.cc:1397] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260926 20:14:33.064635 10460 tablet_bootstrap.cc:492] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9: No bootstrap required, opened a new log
I20260926 20:14:33.064826 10460 ts_tablet_manager.cc:1397] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:14:33.066591 10461 tablet_bootstrap.cc:654] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:33.067420 10461 log.cc:824] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:33.068512 10462 raft_consensus.cc:348] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2af88f2684b10ae72412facb8af38" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44877 } } peers { permanent_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41923 } } peers { permanent_uuid: "0ae334066d1247e3beda763b40e715ee" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34931 } }
I20260926 20:14:33.068753 10462 raft_consensus.cc:374] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:33.068813 10462 raft_consensus.cc:729] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ae334066d1247e3beda763b40e715ee, State: Initialized, Role: FOLLOWER
I20260926 20:14:33.068919 10460 raft_consensus.cc:348] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2af88f2684b10ae72412facb8af38" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44877 } } peers { permanent_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41923 } } peers { permanent_uuid: "0ae334066d1247e3beda763b40e715ee" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34931 } }
I20260926 20:14:33.069178 10460 raft_consensus.cc:374] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:33.069228 10460 raft_consensus.cc:729] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c27f9d635c47d9a14c3c095c3c03d9, State: Initialized, Role: FOLLOWER
I20260926 20:14:33.069286 10462 consensus_queue.cc:260] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2af88f2684b10ae72412facb8af38" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44877 } } peers { permanent_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41923 } } peers { permanent_uuid: "0ae334066d1247e3beda763b40e715ee" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34931 } }
I20260926 20:14:33.069454 10461 tablet_bootstrap.cc:492] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38: No bootstrap required, opened a new log
I20260926 20:14:33.069634 10461 ts_tablet_manager.cc:1397] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260926 20:14:33.069669 10460 consensus_queue.cc:260] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2af88f2684b10ae72412facb8af38" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44877 } } peers { permanent_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41923 } } peers { permanent_uuid: "0ae334066d1247e3beda763b40e715ee" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34931 } }
I20260926 20:14:33.070317 10462 ts_tablet_manager.cc:1428] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260926 20:14:33.070664 10460 ts_tablet_manager.cc:1428] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 20:14:33.070685 10443 heartbeater.cc:503] Master 127.23.50.190:46707 was elected leader, sending a full tablet report...
I20260926 20:14:33.071014 10312 heartbeater.cc:503] Master 127.23.50.190:46707 was elected leader, sending a full tablet report...
I20260926 20:14:33.074941 10461 raft_consensus.cc:348] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2af88f2684b10ae72412facb8af38" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44877 } } peers { permanent_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41923 } } peers { permanent_uuid: "0ae334066d1247e3beda763b40e715ee" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34931 } }
I20260926 20:14:33.075254 10461 raft_consensus.cc:374] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:33.075322 10461 raft_consensus.cc:729] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db2af88f2684b10ae72412facb8af38, State: Initialized, Role: FOLLOWER
I20260926 20:14:33.075717 10461 consensus_queue.cc:260] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2af88f2684b10ae72412facb8af38" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44877 } } peers { permanent_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41923 } } peers { permanent_uuid: "0ae334066d1247e3beda763b40e715ee" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34931 } }
I20260926 20:14:33.076351 10181 heartbeater.cc:503] Master 127.23.50.190:46707 was elected leader, sending a full tablet report...
I20260926 20:14:33.076673 10461 ts_tablet_manager.cc:1428] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 20:14:33.094192 10467 raft_consensus.cc:482] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:14:33.094457 10467 raft_consensus.cc:504] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2af88f2684b10ae72412facb8af38" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44877 } } peers { permanent_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41923 } } peers { permanent_uuid: "0ae334066d1247e3beda763b40e715ee" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34931 } }
I20260926 20:14:33.097621 10467 leader_election.cc:288] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db2af88f2684b10ae72412facb8af38 (127.23.50.129:44877), 0ae334066d1247e3beda763b40e715ee (127.23.50.131:34931)
I20260926 20:14:33.101333 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.096994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:59997 (local address 127.23.50.129:44877)
0926 20:14:33.097128 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:33.097132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:33.097145 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:14:33.097199 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:33.097201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:33.097249 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:33.097351 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:33.097643 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.098273 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.099932 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.100705 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.100828 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.100879 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:33.101109 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 20:14:33.101183 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:14:33.101338 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.096547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:44877 (local address 127.23.50.130:59997)
0926 20:14:33.096694 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:33.096710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:33.096798 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:33.097361 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:33.097364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:33.097380 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:33.097507 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:33.097515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.098399 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.098403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:33.099742 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:33.099753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.100833 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.100837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:33.100962 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.100996 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 20:14:33.101082 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 20:14:33.101165 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":40}
I20260926 20:14:33.102020 10135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3851701784e1da0bb2f5ef78c3d50" candidate_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db2af88f2684b10ae72412facb8af38" is_pre_election: true
I20260926 20:14:33.102169 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.096712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:34931 (local address 127.23.50.130:41267)
0926 20:14:33.097043 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:33.097058 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:33.097146 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:33.097808 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:33.097814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:33.097836 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:33.098015 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:33.098026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.099208 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.099213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:33.100529 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:33.100543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.101665 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.101670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:33.101790 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.101819 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 20:14:33.101959 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 20:14:33.102030 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260926 20:14:33.102160 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.097127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:41267 (local address 127.23.50.131:34931)
0926 20:14:33.097265 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:33.097269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:33.097286 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:14:33.097362 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:33.097367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:33.097426 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:33.097512 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:33.098177 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.099043 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.100673 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.101244 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.101364 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.101416 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:33.101981 (+ 565us) server_negotiation.cc:299] Negotiation successful
0926 20:14:33.102056 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:14:33.102380 10135 raft_consensus.cc:2454] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c27f9d635c47d9a14c3c095c3c03d9 in term 0.
I20260926 20:14:33.102734 10397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3851701784e1da0bb2f5ef78c3d50" candidate_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae334066d1247e3beda763b40e715ee" is_pre_election: true
I20260926 20:14:33.102938 10200 leader_election.cc:302] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db2af88f2684b10ae72412facb8af38, 65c27f9d635c47d9a14c3c095c3c03d9; no voters:
I20260926 20:14:33.103063 10397 raft_consensus.cc:2454] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c27f9d635c47d9a14c3c095c3c03d9 in term 0.
I20260926 20:14:33.103291 10467 raft_consensus.cc:2781] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:14:33.103386 10467 raft_consensus.cc:482] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:14:33.103443 10467 raft_consensus.cc:3037] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:33.104837 10467 raft_consensus.cc:504] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2af88f2684b10ae72412facb8af38" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44877 } } peers { permanent_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41923 } } peers { permanent_uuid: "0ae334066d1247e3beda763b40e715ee" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34931 } }
I20260926 20:14:33.105263 10467 leader_election.cc:288] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [CANDIDATE]: Term 1 election: Requested vote from peers 0db2af88f2684b10ae72412facb8af38 (127.23.50.129:44877), 0ae334066d1247e3beda763b40e715ee (127.23.50.131:34931)
I20260926 20:14:33.105616 10135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3851701784e1da0bb2f5ef78c3d50" candidate_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db2af88f2684b10ae72412facb8af38"
I20260926 20:14:33.105736 10135 raft_consensus.cc:3037] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:33.106279 10397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02d3851701784e1da0bb2f5ef78c3d50" candidate_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ae334066d1247e3beda763b40e715ee"
I20260926 20:14:33.106405 10397 raft_consensus.cc:3037] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:33.107151 10135 raft_consensus.cc:2454] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c27f9d635c47d9a14c3c095c3c03d9 in term 1.
I20260926 20:14:33.107523 10200 leader_election.cc:302] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db2af88f2684b10ae72412facb8af38, 65c27f9d635c47d9a14c3c095c3c03d9; no voters:
W20260926 20:14:33.107640 10182 tablet.cc:2369] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:14:33.107717 10467 raft_consensus.cc:2781] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:33.107733 10397 raft_consensus.cc:2454] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c27f9d635c47d9a14c3c095c3c03d9 in term 1.
I20260926 20:14:33.107930 10467 raft_consensus.cc:686] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 1 LEADER]: Becoming Leader. State: Replica: 65c27f9d635c47d9a14c3c095c3c03d9, State: Running, Role: LEADER
I20260926 20:14:33.108266 10467 consensus_queue.cc:237] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2af88f2684b10ae72412facb8af38" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44877 } } peers { permanent_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41923 } } peers { permanent_uuid: "0ae334066d1247e3beda763b40e715ee" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34931 } }
I20260926 20:14:33.111756 9992 catalog_manager.cc:5495] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65c27f9d635c47d9a14c3c095c3c03d9 (127.23.50.130). New cstate: current_term: 1 leader_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db2af88f2684b10ae72412facb8af38" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65c27f9d635c47d9a14c3c095c3c03d9" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 41923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ae334066d1247e3beda763b40e715ee" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34931 } health_report { overall_health: UNKNOWN } } }
I20260926 20:14:33.136919 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.132823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.23.50.130:41923)
0926 20:14:33.132960 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:33.132963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:33.132976 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:14:33.133045 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:33.133048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:33.133101 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:33.133191 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:14:33.133503 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.134093 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.135417 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:33.135626 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:33.135708 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:33.136097 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:14:33.136500 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:14:33.136611 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:33.136690 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 20:14:33.136724 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 20:14:33.143005 10397 raft_consensus.cc:1261] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee [term 1 FOLLOWER]: Refusing update from remote peer 65c27f9d635c47d9a14c3c095c3c03d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:14:33.143028 10135 raft_consensus.cc:1261] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38 [term 1 FOLLOWER]: Refusing update from remote peer 65c27f9d635c47d9a14c3c095c3c03d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:14:33.143756 10470 consensus_queue.cc:1035] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae334066d1247e3beda763b40e715ee" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 34931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:14:33.143923 10467 consensus_queue.cc:1035] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db2af88f2684b10ae72412facb8af38" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 44877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:14:33.153282 10474 mvcc.cc:205] Tried to move back new op lower bound from 7333698245185945600 to 7333698245054410752. Current Snapshot: MvccSnapshot[applied={T|T < 7333698245185945600}]
I20260926 20:14:33.160892 9992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:14:33.161044 9992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:14:33.163472 9992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 709c911ff5eb4ca8af7cb34485e964fb: Sending DeleteTablet for 3 replicas of tablet 02d3851701784e1da0bb2f5ef78c3d50
I20260926 20:14:33.164230 10115 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3851701784e1da0bb2f5ef78c3d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:33 UTC) from {username='slave'} at 127.0.0.1:52996
I20260926 20:14:33.164307 10246 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3851701784e1da0bb2f5ef78c3d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:33 UTC) from {username='slave'} at 127.0.0.1:58600
I20260926 20:14:33.164461 10377 tablet_service.cc:1514] Processing DeleteTablet for tablet 02d3851701784e1da0bb2f5ef78c3d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:33 UTC) from {username='slave'} at 127.0.0.1:42254
I20260926 20:14:33.164798 10482 tablet_replica.cc:330] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38: stopping tablet replica
I20260926 20:14:33.164966 10484 tablet_replica.cc:330] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee: stopping tablet replica
I20260926 20:14:33.165143 10482 raft_consensus.cc:2229] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:14:33.165417 10483 tablet_replica.cc:330] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9: stopping tablet replica
I20260926 20:14:33.165488 10482 raft_consensus.cc:2258] T 02d3851701784e1da0bb2f5ef78c3d50 P 0db2af88f2684b10ae72412facb8af38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:33.165546 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 10053
I20260926 20:14:33.165692 10483 raft_consensus.cc:2229] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:14:33.166018 10483 raft_consensus.cc:2258] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:33.167801 10484 raft_consensus.cc:2229] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:14:33.168149 10484 raft_consensus.cc:2258] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:33.169551 10483 ts_tablet_manager.cc:1905] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:14:33.169999 10484 ts_tablet_manager.cc:1905] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:14:33.172751 10484 ts_tablet_manager.cc:1918] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:14:33.172861 10484 log.cc:1196] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/02d3851701784e1da0bb2f5ef78c3d50
I20260926 20:14:33.173164 10484 ts_tablet_manager.cc:1939] T 02d3851701784e1da0bb2f5ef78c3d50 P 0ae334066d1247e3beda763b40e715ee: Deleting consensus metadata
I20260926 20:14:33.173439 10483 ts_tablet_manager.cc:1918] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 20:14:33.173440 9978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:44877 (error 108)
I20260926 20:14:33.173614 10483 log.cc:1196] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/02d3851701784e1da0bb2f5ef78c3d50
I20260926 20:14:33.174053 10483 ts_tablet_manager.cc:1939] T 02d3851701784e1da0bb2f5ef78c3d50 P 65c27f9d635c47d9a14c3c095c3c03d9: Deleting consensus metadata
I20260926 20:14:33.174324 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 10184
I20260926 20:14:33.175727 9979 catalog_manager.cc:4841] TS 0ae334066d1247e3beda763b40e715ee (127.23.50.131:34931): tablet 02d3851701784e1da0bb2f5ef78c3d50 (table test-table [id=16a6b444f3f947beb5c255125d27103a]) successfully deleted
I20260926 20:14:33.175741 10457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.175364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:44877 (local address 127.0.0.1:53012)
0926 20:14:33.175560 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:33.175637 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:44877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260926 20:14:33.176451 9978 catalog_manager.cc:4584] TS 0db2af88f2684b10ae72412facb8af38 (127.23.50.129:44877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02d3851701784e1da0bb2f5ef78c3d50: Network error: Client connection negotiation failed: client connection to 127.23.50.129:44877: connect: Connection refused (error 111)
I20260926 20:14:33.180624 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 10315
I20260926 20:14:33.182127 10457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:33.180803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:41923 (local address 127.0.0.1:58628)
0926 20:14:33.181794 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:33.182014 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.130:41923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":884}
I20260926 20:14:33.186894 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 9962
2026-09-26T20:14:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1086 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:33 dist-test-slave-7034 krb5kdc[10488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:33 dist-test-slave-7034 krb5kdc[10488](info): set up 2 sockets
Sep 26 20:14:33 dist-test-slave-7034 krb5kdc[10488](info): commencing operation
krb5kdc: starting...
W20260926 20:14:34.230525 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:34 dist-test-slave-7034 krb5kdc[10488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:14:34.356374 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:45967
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:45967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:34.466658 10501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:34.466918 10501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:34.466958 10501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:34.471701 10501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:34.471763 10501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:34.471784 10501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:34.471801 10501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:34.471844 10501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:34.477392 10501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:45967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:45967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:34.478596 10501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:34.479683 10501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:34.487883 10506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:34.487905 10509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:34.487948 10507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:34.488067 10501 server_base.cc:1034] running on GCE node
I20260926 20:14:34.488723 10501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:34.489928 10501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:34.491117 10501 hybrid_clock.cc:648] HybridClock initialized: now 1790453674491105 us; error 30 us; skew 500 ppm
Sep 26 20:14:34 dist-test-slave-7034 krb5kdc[10488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453674, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:34.493834 10501 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:14:34.494704 10501 webserver.cc:460] Webserver started at http://127.23.50.190:35153/ using document root <none> and password file <none>
I20260926 20:14:34.495261 10501 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:34.495353 10501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:34.495579 10501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:34.497553 10501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "278e42d9d7e84b33b29aee11ed509b5e"
format_stamp: "Formatted at 2026-09-26 20:14:34 on dist-test-slave-7034"
server_key: "475833cff0e2335c54bb5e5774680fe1"
server_key_iv: "5a9d6c2137094b9cf33cab6f50bc7ddd"
server_key_version: "encryptionkey@0"
I20260926 20:14:34.498118 10501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "278e42d9d7e84b33b29aee11ed509b5e"
format_stamp: "Formatted at 2026-09-26 20:14:34 on dist-test-slave-7034"
server_key: "475833cff0e2335c54bb5e5774680fe1"
server_key_iv: "5a9d6c2137094b9cf33cab6f50bc7ddd"
server_key_version: "encryptionkey@0"
I20260926 20:14:34.501700 10501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 20:14:34.504154 10516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:34.505391 10501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:14:34.505537 10501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "278e42d9d7e84b33b29aee11ed509b5e"
format_stamp: "Formatted at 2026-09-26 20:14:34 on dist-test-slave-7034"
server_key: "475833cff0e2335c54bb5e5774680fe1"
server_key_iv: "5a9d6c2137094b9cf33cab6f50bc7ddd"
server_key_version: "encryptionkey@0"
I20260926 20:14:34.505654 10501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:34.516283 10501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:34.519160 10501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:34.519361 10501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:34.526249 10501 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:45967
I20260926 20:14:34.526271 10568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:45967 every 8 connection(s)
I20260926 20:14:34.527189 10501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:34.529973 10569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:34.532519 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 10501
I20260926 20:14:34.532692 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:34.533025 23754 external_mini_cluster.cc:1428] Setting key 6d7219e5dac819767e91747d5e4225cb
I20260926 20:14:34.535806 10569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e: Bootstrap starting.
I20260926 20:14:34.538480 10569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:34.539382 10569 log.cc:824] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:34.541224 10569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e: No bootstrap required, opened a new log
Sep 26 20:14:34 dist-test-slave-7034 krb5kdc[10488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:34.544917 10569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278e42d9d7e84b33b29aee11ed509b5e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45967 } }
I20260926 20:14:34.545208 10569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:34.545274 10569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 278e42d9d7e84b33b29aee11ed509b5e, State: Initialized, Role: FOLLOWER
I20260926 20:14:34.545805 10569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278e42d9d7e84b33b29aee11ed509b5e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45967 } }
I20260926 20:14:34.546029 10569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:34.546118 10569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:34.546216 10569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:34.547324 10569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278e42d9d7e84b33b29aee11ed509b5e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45967 } }
I20260926 20:14:34.547665 10569 leader_election.cc:302] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 278e42d9d7e84b33b29aee11ed509b5e; no voters:
I20260926 20:14:34.547981 10569 leader_election.cc:288] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:34.548103 10574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:34.548106 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:34.534395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.23.50.190:45967)
0926 20:14:34.534806 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:34.534821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:34.534856 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:14:34.535456 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:34.535471 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:34.535798 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:34.536115 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:34.536975 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:34.538048 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:34.539221 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:34.539742 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:34.539925 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:34.542723 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:34.542728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:34.542741 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:34.542767 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:34.544976 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:34.545482 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:34.545486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:34.545491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:34.545597 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:34.545912 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:34.545914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:34.545915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:34.546457 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:34.546599 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:34.546899 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 20:14:34.547201 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":129,"threads_started":1}
I20260926 20:14:34.548385 10574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [term 1 LEADER]: Becoming Leader. State: Replica: 278e42d9d7e84b33b29aee11ed509b5e, State: Running, Role: LEADER
I20260926 20:14:34.548687 10574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278e42d9d7e84b33b29aee11ed509b5e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45967 } }
I20260926 20:14:34.549238 10569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:34.549883 10575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "278e42d9d7e84b33b29aee11ed509b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278e42d9d7e84b33b29aee11ed509b5e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45967 } } }
I20260926 20:14:34.549997 10575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:34.550217 10576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 278e42d9d7e84b33b29aee11ed509b5e. Latest consensus state: current_term: 1 leader_uuid: "278e42d9d7e84b33b29aee11ed509b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "278e42d9d7e84b33b29aee11ed509b5e" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45967 } } }
I20260926 20:14:34.550292 10576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:34.550747 10580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:34.554023 10580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:34.559571 10580 catalog_manager.cc:1347] Generated new cluster ID: dc20293341bc4c6cbb6270280912e319
I20260926 20:14:34.559648 10580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:34.576956 10580 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:34.578318 10580 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:34.593147 10580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e: Generated new TSK 0
I20260926 20:14:34.594099 10580 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.50.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
Entry for principal kudu/127.23.50.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab.
I20260926 20:14:34.627276 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45967
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:34.739634 10595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:34.740037 10595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:34.740126 10595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:34.744338 10595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:34.744431 10595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:34.744527 10595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:14:34.750273 10595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.129.keytab
--principal=kudu/127.23.50.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:34.751652 10595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:34.752743 10595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:34.762766 10600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:34.762766 10603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:34.762812 10601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:34.763159 10595 server_base.cc:1034] running on GCE node
I20260926 20:14:34.763593 10595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:34.764254 10595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:34.765437 10595 hybrid_clock.cc:648] HybridClock initialized: now 1790453674765420 us; error 27 us; skew 500 ppm
Sep 26 20:14:34 dist-test-slave-7034 krb5kdc[10488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453674, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:34.768270 10595 init.cc:377] Logged in from keytab as kudu/127.23.50.129@KRBTEST.COM (short username kudu)
I20260926 20:14:34.769122 10595 webserver.cc:460] Webserver started at http://127.23.50.129:43985/ using document root <none> and password file <none>
I20260926 20:14:34.769687 10595 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:34.769779 10595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:34.770040 10595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:34.772039 10595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "88e94221475a45c1b111c408916c8e4b"
format_stamp: "Formatted at 2026-09-26 20:14:34 on dist-test-slave-7034"
server_key: "f283d07c4e671afdffd2bf04562829f1"
server_key_iv: "c14141cb86efecf37ca4521cf50509c0"
server_key_version: "encryptionkey@0"
I20260926 20:14:34.772599 10595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "88e94221475a45c1b111c408916c8e4b"
format_stamp: "Formatted at 2026-09-26 20:14:34 on dist-test-slave-7034"
server_key: "f283d07c4e671afdffd2bf04562829f1"
server_key_iv: "c14141cb86efecf37ca4521cf50509c0"
server_key_version: "encryptionkey@0"
I20260926 20:14:34.776237 10595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 20:14:34.778630 10610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:34.779868 10595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:34.780009 10595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "88e94221475a45c1b111c408916c8e4b"
format_stamp: "Formatted at 2026-09-26 20:14:34 on dist-test-slave-7034"
server_key: "f283d07c4e671afdffd2bf04562829f1"
server_key_iv: "c14141cb86efecf37ca4521cf50509c0"
server_key_version: "encryptionkey@0"
I20260926 20:14:34.780123 10595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:34.796177 10595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:34.799564 10595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:34.799777 10595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:34.800388 10595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:34.801235 10595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:34.801306 10595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:34.801378 10595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:34.801429 10595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:34.810527 10595 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:41585
I20260926 20:14:34.810549 10723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:41585 every 8 connection(s)
I20260926 20:14:34.811419 10595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:14:34.814201 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 10595
I20260926 20:14:34.814354 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:14:34.814664 23754 external_mini_cluster.cc:1428] Setting key d8a9fa56644d30d7d5f8952e7c0203db
Sep 26 20:14:34 dist-test-slave-7034 krb5kdc[10488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453674, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.129@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.50.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.130@KRBTEST.COM" created.
I20260926 20:14:34.827076 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:34.813356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:48625 (local address 127.23.50.190:45967)
0926 20:14:34.813505 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:34.813509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:34.814561 (+ 1052us) server_negotiation.cc:407] Connection header received
0926 20:14:34.816072 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:34.816076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:34.816132 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:34.816251 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:34.817461 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:34.818119 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:34.819768 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:34.820026 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:34.820115 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:34.823151 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:34.823155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:34.823157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:34.823182 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:34.824721 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:34.825281 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:34.825284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:34.825286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:34.825380 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:34.825754 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:34.825759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:34.825761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:34.826016 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:34.826154 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:34.826741 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 20:14:34.826885 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 20:14:34.827901 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:34.813814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:45967 (local address 127.23.50.129:48625)
0926 20:14:34.814382 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:34.814437 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:34.815847 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:34.816434 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:34.816442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:34.816806 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:34.817305 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:34.817320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:34.818263 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:34.818266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:34.819618 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:34.819633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:34.820201 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:34.820208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:34.820456 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:34.821124 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:14:34.821149 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:34.822960 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:34.824853 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:34.824859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:34.824877 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:34.825146 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:34.825494 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:34.825498 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:34.825500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:34.825654 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:34.826125 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:34.826132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:34.826459 (+ 327us) client_negotiation.cc:769] Sending connection context
0926 20:14:34.826664 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 20:14:34.827018 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":125,"threads_started":1}
I20260926 20:14:34.829326 10724 heartbeater.cc:347] Connected to a master server at 127.23.50.190:45967
I20260926 20:14:34.829574 10724 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:34.830222 10724 heartbeater.cc:511] Master 127.23.50.190:45967 requested a full tablet report, sending...
I20260926 20:14:34.832096 10533 ts_manager.cc:194] Registered new tserver with Master: 88e94221475a45c1b111c408916c8e4b (127.23.50.129:41585)
I20260926 20:14:34.833988 10533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.129@KRBTEST.COM'} at 127.23.50.129:48625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
Entry for principal kudu/127.23.50.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab.
I20260926 20:14:34.844288 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45967
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:34.958067 10729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:34.958367 10729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:34.958410 10729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:34.963214 10729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:34.963279 10729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:34.963348 10729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:14:34.968801 10729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.130.keytab
--principal=kudu/127.23.50.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:34.969825 10729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:34.970933 10729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:34.980504 10735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:34.980572 10737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:34.980602 10729 server_base.cc:1034] running on GCE node
W20260926 20:14:34.980705 10734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:34.981190 10729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:34.981892 10729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:34.983078 10729 hybrid_clock.cc:648] HybridClock initialized: now 1790453674983064 us; error 29 us; skew 500 ppm
Sep 26 20:14:34 dist-test-slave-7034 krb5kdc[10488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453674, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:34.985816 10729 init.cc:377] Logged in from keytab as kudu/127.23.50.130@KRBTEST.COM (short username kudu)
I20260926 20:14:34.986788 10729 webserver.cc:460] Webserver started at http://127.23.50.130:46749/ using document root <none> and password file <none>
I20260926 20:14:34.987372 10729 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:34.987463 10729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:34.987695 10729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:34.989729 10729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "25f53e047b8a44c691ac1ac26382b997"
format_stamp: "Formatted at 2026-09-26 20:14:34 on dist-test-slave-7034"
server_key: "f0ee69f33a0cf067c825a19f8ba15067"
server_key_iv: "e91b269af007a97341b9dbe881070b78"
server_key_version: "encryptionkey@0"
I20260926 20:14:34.990370 10729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "25f53e047b8a44c691ac1ac26382b997"
format_stamp: "Formatted at 2026-09-26 20:14:34 on dist-test-slave-7034"
server_key: "f0ee69f33a0cf067c825a19f8ba15067"
server_key_iv: "e91b269af007a97341b9dbe881070b78"
server_key_version: "encryptionkey@0"
I20260926 20:14:34.994053 10729 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:14:34.996414 10744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:34.997654 10729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:14:34.997813 10729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "25f53e047b8a44c691ac1ac26382b997"
format_stamp: "Formatted at 2026-09-26 20:14:34 on dist-test-slave-7034"
server_key: "f0ee69f33a0cf067c825a19f8ba15067"
server_key_iv: "e91b269af007a97341b9dbe881070b78"
server_key_version: "encryptionkey@0"
I20260926 20:14:34.997990 10729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:35.020435 10729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:35.023516 10729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:35.023732 10729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:35.024356 10729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:35.025210 10729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:35.025292 10729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:35.025388 10729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:35.025437 10729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:35.035015 10729 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:36689
I20260926 20:14:35.035045 10857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:36689 every 8 connection(s)
I20260926 20:14:35.035904 10729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:14:35.041145 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 10729
I20260926 20:14:35.041282 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:14:35.041601 23754 external_mini_cluster.cc:1428] Setting key dac443d91026da4de20f8bb5a18b7a4d
Sep 26 20:14:35 dist-test-slave-7034 krb5kdc[10488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453674, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.130@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:35.050516 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.037642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:46711 (local address 127.23.50.190:45967)
0926 20:14:35.037775 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:35.037779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:35.038653 (+ 874us) server_negotiation.cc:407] Connection header received
0926 20:14:35.039941 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:35.039944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:35.039988 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:35.040061 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:35.041330 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.042116 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.043436 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.043831 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.043957 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.046664 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:35.046667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:35.046669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:35.046692 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:35.048311 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:35.048835 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:35.048843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:35.048845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:35.048917 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:35.049266 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:35.049269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:35.049271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:35.049452 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:35.049560 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:35.050174 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 20:14:35.050319 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:14:35.051410 10860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.038077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:45967 (local address 127.23.50.130:46711)
0926 20:14:35.038495 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:35.038537 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:35.039737 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:35.040244 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:35.040251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:35.040602 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:35.041103 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.041117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.042272 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.042275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.043291 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.043302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.043979 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.043982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.044125 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.044778 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:14:35.044799 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:35.046488 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:35.048432 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:35.048438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:35.048453 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:35.048715 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:35.049008 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:35.049012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:35.049014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:35.049159 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:35.049560 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:35.049568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:35.049892 (+ 324us) client_negotiation.cc:769] Sending connection context
0926 20:14:35.050101 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 20:14:35.050443 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.23.50.131@KRBTEST.COM; defaulting to no policy
I20260926 20:14:35.052615 10858 heartbeater.cc:347] Connected to a master server at 127.23.50.190:45967
I20260926 20:14:35.052910 10858 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:35.053442 10858 heartbeater.cc:511] Master 127.23.50.190:45967 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.131@KRBTEST.COM" created.
I20260926 20:14:35.054673 10533 ts_manager.cc:194] Registered new tserver with Master: 25f53e047b8a44c691ac1ac26382b997 (127.23.50.130:36689)
I20260926 20:14:35.055500 10533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.130@KRBTEST.COM'} at 127.23.50.130:46711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
Entry for principal kudu/127.23.50.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab.
I20260926 20:14:35.072311 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45967
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:35.185551 10863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:35.185871 10863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:35.185920 10863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:35.190117 10863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:35.190181 10863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:35.190277 10863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:14:35.195777 10863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.131.keytab
--principal=kudu/127.23.50.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:35.196939 10863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:35.198063 10863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:35.208515 10871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:35.208503 10868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:35.208745 10863 server_base.cc:1034] running on GCE node
W20260926 20:14:35.208946 10869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:35.209405 10863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:35.210166 10863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:35.211357 10863 hybrid_clock.cc:648] HybridClock initialized: now 1790453675211341 us; error 40 us; skew 500 ppm
Sep 26 20:14:35 dist-test-slave-7034 krb5kdc[10488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453675, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:35.214123 10863 init.cc:377] Logged in from keytab as kudu/127.23.50.131@KRBTEST.COM (short username kudu)
I20260926 20:14:35.215086 10863 webserver.cc:460] Webserver started at http://127.23.50.131:36723/ using document root <none> and password file <none>
I20260926 20:14:35.215684 10863 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:35.215780 10863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:35.216004 10863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:35.218243 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "0af853e29a89429d8770474d1c7f080e"
format_stamp: "Formatted at 2026-09-26 20:14:35 on dist-test-slave-7034"
server_key: "2a2f13279ba29cadfa42cf935e151080"
server_key_iv: "e1c170e0a42e624c56a2a69687d91947"
server_key_version: "encryptionkey@0"
I20260926 20:14:35.218847 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "0af853e29a89429d8770474d1c7f080e"
format_stamp: "Formatted at 2026-09-26 20:14:35 on dist-test-slave-7034"
server_key: "2a2f13279ba29cadfa42cf935e151080"
server_key_iv: "e1c170e0a42e624c56a2a69687d91947"
server_key_version: "encryptionkey@0"
I20260926 20:14:35.222725 10863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:14:35.225152 10878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:35.226511 10863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:35.226655 10863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "0af853e29a89429d8770474d1c7f080e"
format_stamp: "Formatted at 2026-09-26 20:14:35 on dist-test-slave-7034"
server_key: "2a2f13279ba29cadfa42cf935e151080"
server_key_iv: "e1c170e0a42e624c56a2a69687d91947"
server_key_version: "encryptionkey@0"
I20260926 20:14:35.226775 10863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:35.250084 10863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:35.253175 10863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:35.253391 10863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:35.254061 10863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:35.255004 10863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:35.255052 10863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:35.255131 10863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:35.255187 10863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:35.264268 10863 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:42967
I20260926 20:14:35.264293 10991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:42967 every 8 connection(s)
I20260926 20:14:35.265214 10863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:14:35.268684 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 10863
I20260926 20:14:35.268831 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:14:35.269191 23754 external_mini_cluster.cc:1428] Setting key 0005390db188b687d068e5b9743f3aaa
Sep 26 20:14:35 dist-test-slave-7034 krb5kdc[10488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453675, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.131@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:35.280040 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.267114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:39599 (local address 127.23.50.190:45967)
0926 20:14:35.267317 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:35.267321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:35.268233 (+ 912us) server_negotiation.cc:407] Connection header received
0926 20:14:35.270142 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:35.270146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:35.270203 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:35.270296 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:35.271919 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.272490 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.273653 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.273908 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.273995 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.276372 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:35.276383 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:35.276386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:35.276410 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:35.277991 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:35.278522 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:35.278526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:35.278528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:35.278609 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:35.278967 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:35.278972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:35.278974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:35.279159 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:35.279267 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:35.279760 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 20:14:35.279876 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 20:14:35.280874 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.267428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:45967 (local address 127.23.50.131:39599)
0926 20:14:35.267993 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:35.268124 (+ 131us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:35.269933 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:35.270506 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:35.270513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:35.270880 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:35.271788 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.271810 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.272609 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.272612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.273534 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.273544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.274051 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.274054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.274197 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.274746 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:14:35.274761 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:35.276199 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:35.278118 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:35.278126 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:35.278142 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:35.278413 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:35.278709 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:14:35.278712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:14:35.278714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:14:35.278867 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:14:35.279265 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:35.279273 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:35.279524 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 20:14:35.279721 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 20:14:35.280065 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":103,"threads_started":1}
I20260926 20:14:35.282116 10992 heartbeater.cc:347] Connected to a master server at 127.23.50.190:45967
I20260926 20:14:35.282387 10992 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:35.282892 10992 heartbeater.cc:511] Master 127.23.50.190:45967 requested a full tablet report, sending...
I20260926 20:14:35.283979 10533 ts_manager.cc:194] Registered new tserver with Master: 0af853e29a89429d8770474d1c7f080e (127.23.50.131:42967)
I20260926 20:14:35.284548 10533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.50.131@KRBTEST.COM'} at 127.23.50.131:39599
I20260926 20:14:35.295622 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:14:35.305523 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.297791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.23.50.190:45967)
0926 20:14:35.297968 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:35.297973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:35.298464 (+ 491us) server_negotiation.cc:407] Connection header received
0926 20:14:35.298555 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:35.298557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:35.298600 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:35.298679 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:35.299383 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.299931 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.300766 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.300981 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.301063 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.302631 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:35.302634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:35.302636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:35.302659 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:35.303981 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:35.304361 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:35.304364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:35.304365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:35.304440 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:35.304684 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:35.304686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:35.304687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:35.304911 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:35.305015 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:35.305204 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 20:14:35.305300 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 20:14:35.308506 10533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:14:35.311842 10533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:14:35.326545 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.320224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.23.50.130:36689)
0926 20:14:35.321273 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:35.321288 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:35.321324 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:14:35.321402 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:35.321407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:35.321617 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:35.321780 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:35.322135 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.323357 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.325229 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.325810 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.325976 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.326139 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:35.326273 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 20:14:35.326381 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":146,"threads_started":1}
I20260926 20:14:35.326545 11004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.320082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:36689 (local address 127.0.0.1:37260)
0926 20:14:35.320773 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:35.320824 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:35.320953 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:35.321768 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:35.321774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:35.321801 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:35.321998 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.322007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.323480 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.323483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.325119 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.325125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.325972 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.325976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.326084 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.326121 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:14:35.326233 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 20:14:35.326392 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":122,"spinlock_wait_cycles":896,"thread_start_us":83,"threads_started":1}
I20260926 20:14:35.326860 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.321333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.23.50.131:42967)
0926 20:14:35.321994 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:35.322005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:35.322026 (+ 21us) server_negotiation.cc:407] Connection header received
0926 20:14:35.322103 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:35.322111 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:35.322293 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:35.322458 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:35.322671 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.323570 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.324969 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.325923 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.326038 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.326176 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:35.326611 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 20:14:35.326736 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":165,"threads_started":1}
I20260926 20:14:35.326958 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.320105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:42967 (local address 127.0.0.1:51666)
0926 20:14:35.320896 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:35.320910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:35.321001 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:35.322435 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:35.322438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:35.322455 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:35.322548 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.322557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.323701 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.323704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.324837 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.324847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.326234 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.326238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.326343 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.326363 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 20:14:35.326432 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:14:35.326495 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260926 20:14:35.326558 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.320418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:41585 (local address 127.0.0.1:49744)
0926 20:14:35.320973 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:35.320986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:35.321057 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:35.321819 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:35.321825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:35.321882 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:35.322006 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.322011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.323578 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.323581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.324947 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.324958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.326063 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.326066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.326152 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.326173 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 20:14:35.326226 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 20:14:35.326390 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":64,"thread_start_us":79,"threads_started":1}
I20260926 20:14:35.327790 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.320922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.23.50.129:41585)
0926 20:14:35.321422 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:35.321436 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:35.321455 (+ 19us) server_negotiation.cc:407] Connection header received
0926 20:14:35.321516 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:35.321520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:35.321660 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:35.322175 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:35.322311 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.323396 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.325071 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.325922 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.327331 (+ 1409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.327492 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:35.327544 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 20:14:35.327640 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":100,"threads_started":1}
I20260926 20:14:35.330055 10926 tablet_service.cc:1467] Processing CreateTablet for tablet 21dd0202a4bc4544951c32b3c2fbf2c4 (DEFAULT_TABLE table=test-table [id=02786a01ca0b4d308f793eede750ddf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:35.330101 10792 tablet_service.cc:1467] Processing CreateTablet for tablet 21dd0202a4bc4544951c32b3c2fbf2c4 (DEFAULT_TABLE table=test-table [id=02786a01ca0b4d308f793eede750ddf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:35.330507 10658 tablet_service.cc:1467] Processing CreateTablet for tablet 21dd0202a4bc4544951c32b3c2fbf2c4 (DEFAULT_TABLE table=test-table [id=02786a01ca0b4d308f793eede750ddf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:35.331171 10792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21dd0202a4bc4544951c32b3c2fbf2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:35.331177 10926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21dd0202a4bc4544951c32b3c2fbf2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:35.331386 10658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21dd0202a4bc4544951c32b3c2fbf2c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:35.337183 11009 tablet_bootstrap.cc:492] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e: Bootstrap starting.
I20260926 20:14:35.338666 11010 tablet_bootstrap.cc:492] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b: Bootstrap starting.
I20260926 20:14:35.338666 11011 tablet_bootstrap.cc:492] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997: Bootstrap starting.
I20260926 20:14:35.340005 11009 tablet_bootstrap.cc:654] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:35.340811 11010 tablet_bootstrap.cc:654] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:35.340823 11011 tablet_bootstrap.cc:654] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:35.341354 11009 log.cc:824] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:35.341746 11010 log.cc:824] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:35.342089 11011 log.cc:824] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:35.343416 11009 tablet_bootstrap.cc:492] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e: No bootstrap required, opened a new log
I20260926 20:14:35.343633 11009 ts_tablet_manager.cc:1397] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 20:14:35.344022 11011 tablet_bootstrap.cc:492] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997: No bootstrap required, opened a new log
I20260926 20:14:35.344214 11011 ts_tablet_manager.cc:1397] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 20:14:35.347016 11009 raft_consensus.cc:348] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f53e047b8a44c691ac1ac26382b997" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 36689 } } peers { permanent_uuid: "0af853e29a89429d8770474d1c7f080e" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42967 } } peers { permanent_uuid: "88e94221475a45c1b111c408916c8e4b" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41585 } }
I20260926 20:14:35.347263 11009 raft_consensus.cc:374] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:35.347324 11009 raft_consensus.cc:729] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af853e29a89429d8770474d1c7f080e, State: Initialized, Role: FOLLOWER
I20260926 20:14:35.347472 11011 raft_consensus.cc:348] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f53e047b8a44c691ac1ac26382b997" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 36689 } } peers { permanent_uuid: "0af853e29a89429d8770474d1c7f080e" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42967 } } peers { permanent_uuid: "88e94221475a45c1b111c408916c8e4b" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41585 } }
I20260926 20:14:35.347672 11011 raft_consensus.cc:374] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:35.347748 11011 raft_consensus.cc:729] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f53e047b8a44c691ac1ac26382b997, State: Initialized, Role: FOLLOWER
I20260926 20:14:35.347827 11009 consensus_queue.cc:260] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f53e047b8a44c691ac1ac26382b997" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 36689 } } peers { permanent_uuid: "0af853e29a89429d8770474d1c7f080e" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42967 } } peers { permanent_uuid: "88e94221475a45c1b111c408916c8e4b" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41585 } }
I20260926 20:14:35.348176 11011 consensus_queue.cc:260] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f53e047b8a44c691ac1ac26382b997" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 36689 } } peers { permanent_uuid: "0af853e29a89429d8770474d1c7f080e" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42967 } } peers { permanent_uuid: "88e94221475a45c1b111c408916c8e4b" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41585 } }
I20260926 20:14:35.348591 10992 heartbeater.cc:503] Master 127.23.50.190:45967 was elected leader, sending a full tablet report...
I20260926 20:14:35.348814 10858 heartbeater.cc:503] Master 127.23.50.190:45967 was elected leader, sending a full tablet report...
I20260926 20:14:35.348932 11009 ts_tablet_manager.cc:1428] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 20:14:35.349179 11011 ts_tablet_manager.cc:1428] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 20:14:35.351330 11010 tablet_bootstrap.cc:492] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b: No bootstrap required, opened a new log
I20260926 20:14:35.351541 11010 ts_tablet_manager.cc:1397] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260926 20:14:35.355708 11010 raft_consensus.cc:348] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f53e047b8a44c691ac1ac26382b997" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 36689 } } peers { permanent_uuid: "0af853e29a89429d8770474d1c7f080e" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42967 } } peers { permanent_uuid: "88e94221475a45c1b111c408916c8e4b" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41585 } }
I20260926 20:14:35.355998 11010 raft_consensus.cc:374] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:35.356062 11010 raft_consensus.cc:729] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e94221475a45c1b111c408916c8e4b, State: Initialized, Role: FOLLOWER
I20260926 20:14:35.356478 11010 consensus_queue.cc:260] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f53e047b8a44c691ac1ac26382b997" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 36689 } } peers { permanent_uuid: "0af853e29a89429d8770474d1c7f080e" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42967 } } peers { permanent_uuid: "88e94221475a45c1b111c408916c8e4b" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41585 } }
I20260926 20:14:35.357213 10724 heartbeater.cc:503] Master 127.23.50.190:45967 was elected leader, sending a full tablet report...
I20260926 20:14:35.357534 11010 ts_tablet_manager.cc:1428] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 20:14:35.516352 10993 tablet.cc:2369] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:14:35.536985 10859 tablet.cc:2369] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 20:14:35.562773 10725 tablet.cc:2369] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:14:35.570371 11017 raft_consensus.cc:482] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:14:35.570560 11017 raft_consensus.cc:504] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f53e047b8a44c691ac1ac26382b997" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 36689 } } peers { permanent_uuid: "0af853e29a89429d8770474d1c7f080e" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42967 } } peers { permanent_uuid: "88e94221475a45c1b111c408916c8e4b" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41585 } }
I20260926 20:14:35.571660 11017 leader_election.cc:288] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25f53e047b8a44c691ac1ac26382b997 (127.23.50.130:36689), 0af853e29a89429d8770474d1c7f080e (127.23.50.131:42967)
I20260926 20:14:35.575835 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.572040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:41953 (local address 127.23.50.131:42967)
0926 20:14:35.572172 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:35.572176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:35.572454 (+ 278us) server_negotiation.cc:407] Connection header received
0926 20:14:35.572638 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:35.572647 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:35.572693 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:35.572774 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:35.573101 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.573598 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.574815 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.575214 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.575363 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.575420 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:35.575638 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 20:14:35.575707 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:14:35.575835 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.572125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:35481 (local address 127.23.50.130:36689)
0926 20:14:35.572255 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:35.572260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:35.572472 (+ 212us) server_negotiation.cc:407] Connection header received
0926 20:14:35.572644 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:35.572651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:35.572696 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:35.572784 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:35.573091 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.573598 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.574849 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.575232 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.575364 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.575419 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:35.575623 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:14:35.575693 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:14:35.575858 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.571991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:36689 (local address 127.23.50.129:35481)
0926 20:14:35.572370 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:35.572381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:35.572522 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:35.572795 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:35.572798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:35.572813 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:35.572970 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.572976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.573716 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.573720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.574712 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.574722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.575359 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.575362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.575476 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.575517 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 20:14:35.575605 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 20:14:35.575677 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":88,"spinlock_wait_cycles":24320,"thread_start_us":121,"threads_started":1}
I20260926 20:14:35.575873 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.571982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:42967 (local address 127.23.50.129:41953)
0926 20:14:35.572341 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:35.572353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:35.572519 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:35.572808 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:35.572813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:35.572831 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:35.572982 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.572987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.573710 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.573716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.574684 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:35.574694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.575359 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.575362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:35.575491 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.575533 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 20:14:35.575621 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 20:14:35.575689 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":70,"spinlock_wait_cycles":2432,"thread_start_us":136,"threads_started":1}
I20260926 20:14:35.576483 10812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd0202a4bc4544951c32b3c2fbf2c4" candidate_uuid: "88e94221475a45c1b111c408916c8e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25f53e047b8a44c691ac1ac26382b997" is_pre_election: true
I20260926 20:14:35.576560 10946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd0202a4bc4544951c32b3c2fbf2c4" candidate_uuid: "88e94221475a45c1b111c408916c8e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af853e29a89429d8770474d1c7f080e" is_pre_election: true
I20260926 20:14:35.576815 10812 raft_consensus.cc:2454] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88e94221475a45c1b111c408916c8e4b in term 0.
I20260926 20:14:35.576848 10946 raft_consensus.cc:2454] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88e94221475a45c1b111c408916c8e4b in term 0.
I20260926 20:14:35.577338 10611 leader_election.cc:302] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af853e29a89429d8770474d1c7f080e, 88e94221475a45c1b111c408916c8e4b; no voters:
I20260926 20:14:35.577653 11017 raft_consensus.cc:2781] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:14:35.577747 11017 raft_consensus.cc:482] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:14:35.577809 11017 raft_consensus.cc:3037] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:35.578920 11017 raft_consensus.cc:504] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f53e047b8a44c691ac1ac26382b997" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 36689 } } peers { permanent_uuid: "0af853e29a89429d8770474d1c7f080e" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42967 } } peers { permanent_uuid: "88e94221475a45c1b111c408916c8e4b" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41585 } }
I20260926 20:14:35.579281 11017 leader_election.cc:288] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [CANDIDATE]: Term 1 election: Requested vote from peers 25f53e047b8a44c691ac1ac26382b997 (127.23.50.130:36689), 0af853e29a89429d8770474d1c7f080e (127.23.50.131:42967)
I20260926 20:14:35.579639 10946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd0202a4bc4544951c32b3c2fbf2c4" candidate_uuid: "88e94221475a45c1b111c408916c8e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0af853e29a89429d8770474d1c7f080e"
I20260926 20:14:35.579777 10946 raft_consensus.cc:3037] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:35.580809 10946 raft_consensus.cc:2454] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e94221475a45c1b111c408916c8e4b in term 1.
I20260926 20:14:35.581099 10812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dd0202a4bc4544951c32b3c2fbf2c4" candidate_uuid: "88e94221475a45c1b111c408916c8e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25f53e047b8a44c691ac1ac26382b997"
I20260926 20:14:35.581140 10611 leader_election.cc:302] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0af853e29a89429d8770474d1c7f080e, 88e94221475a45c1b111c408916c8e4b; no voters:
I20260926 20:14:35.581239 10812 raft_consensus.cc:3037] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:35.581327 11017 raft_consensus.cc:2781] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:35.581564 11017 raft_consensus.cc:686] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [term 1 LEADER]: Becoming Leader. State: Replica: 88e94221475a45c1b111c408916c8e4b, State: Running, Role: LEADER
I20260926 20:14:35.581960 11017 consensus_queue.cc:237] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f53e047b8a44c691ac1ac26382b997" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 36689 } } peers { permanent_uuid: "0af853e29a89429d8770474d1c7f080e" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42967 } } peers { permanent_uuid: "88e94221475a45c1b111c408916c8e4b" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41585 } }
I20260926 20:14:35.582547 10812 raft_consensus.cc:2454] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e94221475a45c1b111c408916c8e4b in term 1.
I20260926 20:14:35.585314 10531 catalog_manager.cc:5495] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b reported cstate change: term changed from 0 to 1, leader changed from <none> to 88e94221475a45c1b111c408916c8e4b (127.23.50.129). New cstate: current_term: 1 leader_uuid: "88e94221475a45c1b111c408916c8e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f53e047b8a44c691ac1ac26382b997" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 36689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0af853e29a89429d8770474d1c7f080e" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88e94221475a45c1b111c408916c8e4b" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 41585 } health_report { overall_health: HEALTHY } } }
I20260926 20:14:35.623289 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.619760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.23.50.129:41585)
0926 20:14:35.619906 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:35.619910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:35.619925 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:14:35.620098 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:35.620100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:35.620157 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:35.620276 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:14:35.620543 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.621057 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.621980 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.622196 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.622280 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.622558 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:14:35.622982 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:14:35.623076 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:35.623128 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 20:14:35.623180 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:14:35.629150 10812 raft_consensus.cc:1261] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997 [term 1 FOLLOWER]: Refusing update from remote peer 88e94221475a45c1b111c408916c8e4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:14:35.629150 10946 raft_consensus.cc:1261] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e [term 1 FOLLOWER]: Refusing update from remote peer 88e94221475a45c1b111c408916c8e4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:14:35.629876 11017 consensus_queue.cc:1035] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "25f53e047b8a44c691ac1ac26382b997" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 36689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:14:35.630080 11020 consensus_queue.cc:1035] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0af853e29a89429d8770474d1c7f080e" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:14:35.645833 10531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:14:35.646076 10531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:14:35.648555 10531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 278e42d9d7e84b33b29aee11ed509b5e: Sending DeleteTablet for 3 replicas of tablet 21dd0202a4bc4544951c32b3c2fbf2c4
I20260926 20:14:35.649443 10926 tablet_service.cc:1514] Processing DeleteTablet for tablet 21dd0202a4bc4544951c32b3c2fbf2c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:35 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:51666
I20260926 20:14:35.649442 10792 tablet_service.cc:1514] Processing DeleteTablet for tablet 21dd0202a4bc4544951c32b3c2fbf2c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:35 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:37260
I20260926 20:14:35.649998 11033 tablet_replica.cc:330] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e: stopping tablet replica
I20260926 20:14:35.650431 11033 raft_consensus.cc:2229] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:14:35.650573 10658 tablet_service.cc:1514] Processing DeleteTablet for tablet 21dd0202a4bc4544951c32b3c2fbf2c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:35 UTC) from {username='kudu', principal='kudu/127.23.50.190@KRBTEST.COM'} at 127.0.0.1:49744
I20260926 20:14:35.650985 11033 raft_consensus.cc:2258] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:35.651070 11035 tablet_replica.cc:330] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b: stopping tablet replica
I20260926 20:14:35.651330 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 10595
I20260926 20:14:35.651388 11035 raft_consensus.cc:2229] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 88e94221475a45c1b111c408916c8e4b [term 1 LEADER]: Raft consensus shutting down.
I20260926 20:14:35.654130 11034 tablet_replica.cc:330] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997: stopping tablet replica
I20260926 20:14:35.654430 11034 raft_consensus.cc:2229] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:14:35.654703 11034 raft_consensus.cc:2258] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:35.655263 11033 ts_tablet_manager.cc:1905] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:14:35.655864 11034 ts_tablet_manager.cc:1905] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:14:35.658039 11034 ts_tablet_manager.cc:1918] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:14:35.658167 11034 log.cc:1196] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/21dd0202a4bc4544951c32b3c2fbf2c4
I20260926 20:14:35.658529 11034 ts_tablet_manager.cc:1939] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 25f53e047b8a44c691ac1ac26382b997: Deleting consensus metadata
I20260926 20:14:35.658825 11033 ts_tablet_manager.cc:1918] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:14:35.658923 11033 log.cc:1196] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/21dd0202a4bc4544951c32b3c2fbf2c4
W20260926 20:14:35.658959 10519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:41585 (error 108)
I20260926 20:14:35.659263 11033 ts_tablet_manager.cc:1939] T 21dd0202a4bc4544951c32b3c2fbf2c4 P 0af853e29a89429d8770474d1c7f080e: Deleting consensus metadata
I20260926 20:14:35.659873 10518 catalog_manager.cc:4841] TS 25f53e047b8a44c691ac1ac26382b997 (127.23.50.130:36689): tablet 21dd0202a4bc4544951c32b3c2fbf2c4 (table test-table [id=02786a01ca0b4d308f793eede750ddf5]) successfully deleted
I20260926 20:14:35.660111 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.659772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:41585 (local address 127.0.0.1:49760)
0926 20:14:35.659949 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:35.660019 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:41585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 20:14:35.660254 10519 catalog_manager.cc:4584] TS 88e94221475a45c1b111c408916c8e4b (127.23.50.129:41585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21dd0202a4bc4544951c32b3c2fbf2c4: Network error: Client connection negotiation failed: client connection to 127.23.50.129:41585: connect: Connection refused (error 111)
I20260926 20:14:35.660362 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 10729
I20260926 20:14:35.660480 10517 catalog_manager.cc:4841] TS 0af853e29a89429d8770474d1c7f080e (127.23.50.131:42967): tablet 21dd0202a4bc4544951c32b3c2fbf2c4 (table test-table [id=02786a01ca0b4d308f793eede750ddf5]) successfully deleted
I20260926 20:14:35.666349 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 10863
I20260926 20:14:35.672139 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 10501
2026-09-26T20:14:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2482 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T20:14:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:35Z Disabled control of system clock
I20260926 20:14:35.702700 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:45643
--webserver_interface=127.23.50.190
--webserver_port=0
--builtin_ntp_servers=127.23.50.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:45643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:14:35.813099 11040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:35.813370 11040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:35.813405 11040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:35.817482 11040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:35.817535 11040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:35.817551 11040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:35.817567 11040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:35.817584 11040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:35.822999 11040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:45643
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:45643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:35.824095 11040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:35.825498 11040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:35.833997 11048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:35.834044 11045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:35.834228 11040 server_base.cc:1034] running on GCE node
W20260926 20:14:35.834013 11046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:35.835019 11040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:35.836126 11040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:35.837325 11040 hybrid_clock.cc:648] HybridClock initialized: now 1790453675837404 us; error 119 us; skew 500 ppm
I20260926 20:14:35.839257 11040 webserver.cc:460] Webserver started at http://127.23.50.190:33309/ using document root <none> and password file <none>
I20260926 20:14:35.839835 11040 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:35.839936 11040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:35.840163 11040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:35.842252 11040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "f5618a6900e9496289a1eb5060bee44d"
format_stamp: "Formatted at 2026-09-26 20:14:35 on dist-test-slave-7034"
server_key: "6cc3a5f49a684f7168fe071f83351dbf"
server_key_iv: "d2536a3a5a15363dabbec44b176944c7"
server_key_version: "encryptionkey@0"
I20260926 20:14:35.842813 11040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "f5618a6900e9496289a1eb5060bee44d"
format_stamp: "Formatted at 2026-09-26 20:14:35 on dist-test-slave-7034"
server_key: "6cc3a5f49a684f7168fe071f83351dbf"
server_key_iv: "d2536a3a5a15363dabbec44b176944c7"
server_key_version: "encryptionkey@0"
I20260926 20:14:35.846423 11040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:14:35.848691 11054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:35.849967 11040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:35.850101 11040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "f5618a6900e9496289a1eb5060bee44d"
format_stamp: "Formatted at 2026-09-26 20:14:35 on dist-test-slave-7034"
server_key: "6cc3a5f49a684f7168fe071f83351dbf"
server_key_iv: "d2536a3a5a15363dabbec44b176944c7"
server_key_version: "encryptionkey@0"
I20260926 20:14:35.850208 11040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:35.874557 11040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:35.875319 11040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:35.875531 11040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:35.882447 11040 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:45643
I20260926 20:14:35.882508 11106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:45643 every 8 connection(s)
I20260926 20:14:35.883440 11040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:35.886555 11107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:35.888626 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 11040
I20260926 20:14:35.888757 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:35.889078 23754 external_mini_cluster.cc:1428] Setting key 46e98fdeb042655b42d42d35a91f3795
I20260926 20:14:35.892759 11107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d: Bootstrap starting.
I20260926 20:14:35.895243 11107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:35.896129 11107 log.cc:824] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:35.898218 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:35.890513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.23.50.190:45643)
0926 20:14:35.891118 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:35.891132 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:35.891166 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:14:35.891805 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:35.891821 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:35.892214 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:35.892487 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:35.892809 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.893807 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.894987 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:35.895518 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:35.895637 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:35.896473 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:35.896478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:35.896490 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:35.896856 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:35.896909 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:35.896928 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:35.897004 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:35.897161 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 20:14:35.897469 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":138,"threads_started":1}
I20260926 20:14:35.898241 11107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d: No bootstrap required, opened a new log
I20260926 20:14:35.902350 11107 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5618a6900e9496289a1eb5060bee44d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45643 } }
I20260926 20:14:35.902570 11107 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:35.902613 11107 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5618a6900e9496289a1eb5060bee44d, State: Initialized, Role: FOLLOWER
I20260926 20:14:35.903082 11107 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5618a6900e9496289a1eb5060bee44d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45643 } }
I20260926 20:14:35.903231 11107 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:35.903297 11107 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:35.903419 11107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:35.904407 11107 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5618a6900e9496289a1eb5060bee44d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45643 } }
I20260926 20:14:35.904736 11107 leader_election.cc:302] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5618a6900e9496289a1eb5060bee44d; no voters:
I20260926 20:14:35.905037 11107 leader_election.cc:288] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:35.905164 11112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:35.905435 11112 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [term 1 LEADER]: Becoming Leader. State: Replica: f5618a6900e9496289a1eb5060bee44d, State: Running, Role: LEADER
I20260926 20:14:35.905758 11112 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5618a6900e9496289a1eb5060bee44d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45643 } }
I20260926 20:14:35.906287 11107 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:35.907470 11114 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5618a6900e9496289a1eb5060bee44d. Latest consensus state: current_term: 1 leader_uuid: "f5618a6900e9496289a1eb5060bee44d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5618a6900e9496289a1eb5060bee44d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45643 } } }
I20260926 20:14:35.907518 11113 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5618a6900e9496289a1eb5060bee44d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5618a6900e9496289a1eb5060bee44d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45643 } } }
I20260926 20:14:35.907606 11114 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:35.907606 11113 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:35.908087 11121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:35.911329 11121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:35.916891 11121 catalog_manager.cc:1347] Generated new cluster ID: 8b9f7f472901434b95532dc7c509ff0f
I20260926 20:14:35.916972 11121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:35.923415 11121 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:35.924603 11121 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:35.934422 11121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d: Generated new TSK 0
I20260926 20:14:35.935245 11121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:35.945386 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.129:0
--local_ip_for_outbound_sockets=127.23.50.129
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45643
--builtin_ntp_servers=127.23.50.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 20:14:36.057119 11131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:36.057430 11131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:36.057507 11131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:36.061550 11131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:36.061635 11131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:36.061720 11131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.129
I20260926 20:14:36.066455 11131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.50.129
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.129
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:36.067566 11131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:36.068975 11131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:36.078583 11139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:36.078732 11131 server_base.cc:1034] running on GCE node
W20260926 20:14:36.078569 11136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:36.078662 11137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:36.079349 11131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:36.079944 11131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:36.081104 11131 hybrid_clock.cc:648] HybridClock initialized: now 1790453676081084 us; error 34 us; skew 500 ppm
I20260926 20:14:36.082945 11131 webserver.cc:460] Webserver started at http://127.23.50.129:43451/ using document root <none> and password file <none>
I20260926 20:14:36.083510 11131 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:36.083572 11131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:36.083809 11131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:36.085809 11131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/data/instance:
uuid: "5713fb460bf44266a6e10d6f052dc38e"
format_stamp: "Formatted at 2026-09-26 20:14:36 on dist-test-slave-7034"
server_key: "566cad798a9781b9600ff7c5f0f39286"
server_key_iv: "3c0dac9c80fac58b993c7fa5997d8789"
server_key_version: "encryptionkey@0"
I20260926 20:14:36.086431 11131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance:
uuid: "5713fb460bf44266a6e10d6f052dc38e"
format_stamp: "Formatted at 2026-09-26 20:14:36 on dist-test-slave-7034"
server_key: "566cad798a9781b9600ff7c5f0f39286"
server_key_iv: "3c0dac9c80fac58b993c7fa5997d8789"
server_key_version: "encryptionkey@0"
I20260926 20:14:36.090101 11131 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 20:14:36.092474 11145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:36.093773 11131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:14:36.093943 11131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/wal
uuid: "5713fb460bf44266a6e10d6f052dc38e"
format_stamp: "Formatted at 2026-09-26 20:14:36 on dist-test-slave-7034"
server_key: "566cad798a9781b9600ff7c5f0f39286"
server_key_iv: "3c0dac9c80fac58b993c7fa5997d8789"
server_key_version: "encryptionkey@0"
I20260926 20:14:36.094074 11131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:36.109421 11131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:36.110147 11131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:36.110349 11131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:36.110912 11131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:36.111829 11131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:36.111898 11131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:36.111965 11131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:36.112012 11131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:36.120615 11131 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.129:45509
I20260926 20:14:36.120637 11258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.129:45509 every 8 connection(s)
I20260926 20:14:36.121515 11131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/data/info.pb
I20260926 20:14:36.121819 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 11131
I20260926 20:14:36.122002 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-0/wal/instance
I20260926 20:14:36.122356 23754 external_mini_cluster.cc:1428] Setting key 7c468753a0bdab934a25ddefdad9b8ac
I20260926 20:14:36.124464 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.130:0
--local_ip_for_outbound_sockets=127.23.50.130
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45643
--builtin_ntp_servers=127.23.50.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:14:36.132529 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.123349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:34529 (local address 127.23.50.190:45643)
0926 20:14:36.123495 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:36.123500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:36.124791 (+ 1291us) server_negotiation.cc:407] Connection header received
0926 20:14:36.126585 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:36.126588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:36.126634 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:36.126709 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:36.127886 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.128519 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.130445 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.130656 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.130736 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.132061 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:36.132064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:36.132066 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:36.132078 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:36.132097 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:36.132105 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:36.132158 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:36.132365 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 20:14:36.132415 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:14:36.133795 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.123917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:45643 (local address 127.23.50.129:34529)
0926 20:14:36.124630 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:36.124682 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:36.126342 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:36.126908 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:36.126919 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:36.127037 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:36.127735 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.127757 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.128666 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.128672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.130322 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.130336 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.130824 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.130832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.131064 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.131730 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:14:36.131735 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:36.131795 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:14:36.131887 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:36.132197 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:36.132209 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:36.132218 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 20:14:36.132375 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 20:14:36.132698 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":124,"threads_started":1}
I20260926 20:14:36.135231 11259 heartbeater.cc:347] Connected to a master server at 127.23.50.190:45643
I20260926 20:14:36.135596 11259 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:36.136165 11259 heartbeater.cc:511] Master 127.23.50.190:45643 requested a full tablet report, sending...
I20260926 20:14:36.137624 11071 ts_manager.cc:194] Registered new tserver with Master: 5713fb460bf44266a6e10d6f052dc38e (127.23.50.129:45509)
I20260926 20:14:36.138823 11071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.50.129:34529
W20260926 20:14:36.242897 11262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:36.243261 11262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:36.243356 11262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:36.247627 11262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:36.247731 11262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:36.247833 11262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.130
I20260926 20:14:36.252889 11262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.50.130
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.130
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:36.254058 11262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:36.255499 11262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:36.265261 11268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:36.265256 11267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:36.265416 11262 server_base.cc:1034] running on GCE node
W20260926 20:14:36.265586 11270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:36.266078 11262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:36.266763 11262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:36.267972 11262 hybrid_clock.cc:648] HybridClock initialized: now 1790453676267954 us; error 30 us; skew 500 ppm
I20260926 20:14:36.270012 11262 webserver.cc:460] Webserver started at http://127.23.50.130:32983/ using document root <none> and password file <none>
I20260926 20:14:36.270709 11262 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:36.270809 11262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:36.271021 11262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:36.273097 11262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/data/instance:
uuid: "f9d0b57cecda4e2da3523a203df5159f"
format_stamp: "Formatted at 2026-09-26 20:14:36 on dist-test-slave-7034"
server_key: "92f80ae179bd908852c6e185028de1ba"
server_key_iv: "8e0281ea5586430a1d81ba77cb1bcb3d"
server_key_version: "encryptionkey@0"
I20260926 20:14:36.273670 11262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance:
uuid: "f9d0b57cecda4e2da3523a203df5159f"
format_stamp: "Formatted at 2026-09-26 20:14:36 on dist-test-slave-7034"
server_key: "92f80ae179bd908852c6e185028de1ba"
server_key_iv: "8e0281ea5586430a1d81ba77cb1bcb3d"
server_key_version: "encryptionkey@0"
I20260926 20:14:36.277407 11262 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:14:36.279974 11276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:36.281206 11262 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:14:36.281340 11262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/wal
uuid: "f9d0b57cecda4e2da3523a203df5159f"
format_stamp: "Formatted at 2026-09-26 20:14:36 on dist-test-slave-7034"
server_key: "92f80ae179bd908852c6e185028de1ba"
server_key_iv: "8e0281ea5586430a1d81ba77cb1bcb3d"
server_key_version: "encryptionkey@0"
I20260926 20:14:36.281458 11262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:36.316768 11262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:36.317539 11262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:36.317754 11262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:36.318442 11262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:36.319439 11262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:36.319512 11262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:36.319586 11262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:36.319634 11262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:36.329077 11262 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.130:42919
I20260926 20:14:36.329094 11389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.130:42919 every 8 connection(s)
I20260926 20:14:36.330122 11262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/data/info.pb
I20260926 20:14:36.331864 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 11262
I20260926 20:14:36.332002 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/wal/instance
I20260926 20:14:36.332355 23754 external_mini_cluster.cc:1428] Setting key b8d220cb5397baa278eccbaf28a7cb90
I20260926 20:14:36.334511 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.50.131:0
--local_ip_for_outbound_sockets=127.23.50.131
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45643
--builtin_ntp_servers=127.23.50.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 20:14:36.340986 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.331997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.130:54281 (local address 127.23.50.190:45643)
0926 20:14:36.332147 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:36.332152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:36.333432 (+ 1280us) server_negotiation.cc:407] Connection header received
0926 20:14:36.335330 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:36.335333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:36.335376 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:36.335443 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:36.336736 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.337479 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.338692 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.339097 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.339245 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.340380 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:36.340387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:36.340393 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:36.340414 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:36.340442 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:36.340457 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:36.340547 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:36.340689 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 20:14:36.340769 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:14:36.341768 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.332517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:45643 (local address 127.23.50.130:54281)
0926 20:14:36.333286 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:36.333334 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:36.335097 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:36.335635 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:36.335646 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:36.335762 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:36.336581 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.336600 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.337615 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.337617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.338565 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.338575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.339267 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.339270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.339437 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.340103 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:14:36.340105 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:36.340147 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:14:36.340215 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:36.340544 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:36.340551 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:36.340555 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 20:14:36.340652 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 20:14:36.340888 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":157,"threads_started":1}
I20260926 20:14:36.343053 11390 heartbeater.cc:347] Connected to a master server at 127.23.50.190:45643
I20260926 20:14:36.343302 11390 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:36.343832 11390 heartbeater.cc:511] Master 127.23.50.190:45643 requested a full tablet report, sending...
I20260926 20:14:36.344889 11071 ts_manager.cc:194] Registered new tserver with Master: f9d0b57cecda4e2da3523a203df5159f (127.23.50.130:42919)
I20260926 20:14:36.345461 11071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.50.130:54281
W20260926 20:14:36.447129 11393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:36.447382 11393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:36.447445 11393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:36.451458 11393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:36.451506 11393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:36.451581 11393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.50.131
I20260926 20:14:36.456414 11393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.50.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.50.131
--webserver_port=0
--tserver_master_addrs=127.23.50.190:45643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.50.131
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:36.457506 11393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:36.458912 11393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:36.468554 11398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:36.468536 11401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:36.468540 11399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:36.468870 11393 server_base.cc:1034] running on GCE node
I20260926 20:14:36.469329 11393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:36.470028 11393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:36.471237 11393 hybrid_clock.cc:648] HybridClock initialized: now 1790453676471212 us; error 37 us; skew 500 ppm
I20260926 20:14:36.473093 11393 webserver.cc:460] Webserver started at http://127.23.50.131:33567/ using document root <none> and password file <none>
I20260926 20:14:36.473687 11393 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:36.473779 11393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:36.474032 11393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:36.476272 11393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/data/instance:
uuid: "eb7ee0a97a844a638923d5f9fffb91df"
format_stamp: "Formatted at 2026-09-26 20:14:36 on dist-test-slave-7034"
server_key: "1fe2c9f2e2784241a23d2b2a36bc0f0e"
server_key_iv: "d92218089949c0dcb0ecc011ea5efb9e"
server_key_version: "encryptionkey@0"
I20260926 20:14:36.477077 11393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance:
uuid: "eb7ee0a97a844a638923d5f9fffb91df"
format_stamp: "Formatted at 2026-09-26 20:14:36 on dist-test-slave-7034"
server_key: "1fe2c9f2e2784241a23d2b2a36bc0f0e"
server_key_iv: "d92218089949c0dcb0ecc011ea5efb9e"
server_key_version: "encryptionkey@0"
I20260926 20:14:36.481393 11393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 20:14:36.483846 11407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:36.485132 11393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:14:36.485249 11393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/wal
uuid: "eb7ee0a97a844a638923d5f9fffb91df"
format_stamp: "Formatted at 2026-09-26 20:14:36 on dist-test-slave-7034"
server_key: "1fe2c9f2e2784241a23d2b2a36bc0f0e"
server_key_iv: "d92218089949c0dcb0ecc011ea5efb9e"
server_key_version: "encryptionkey@0"
I20260926 20:14:36.485405 11393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:36.510751 11393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:36.511394 11393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:36.511605 11393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:36.512138 11393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 20:14:36.513011 11393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 20:14:36.513056 11393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:36.513095 11393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 20:14:36.513110 11393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:36.521652 11393 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.131:38337
I20260926 20:14:36.521701 11520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.131:38337 every 8 connection(s)
I20260926 20:14:36.522554 11393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/data/info.pb
I20260926 20:14:36.532028 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.524349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.131:53875 (local address 127.23.50.190:45643)
0926 20:14:36.524496 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:36.524500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:36.525607 (+ 1107us) server_negotiation.cc:407] Connection header received
0926 20:14:36.527144 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:36.527147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:36.527208 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:36.527326 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:36.528255 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.528799 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.530033 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.530270 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.530390 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.531513 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:36.531517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:36.531520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:36.531534 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:36.531562 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:36.531570 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:36.531650 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:36.531814 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 20:14:36.531880 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:14:36.532143 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 11393
I20260926 20:14:36.532263 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/wal/instance
I20260926 20:14:36.532568 23754 external_mini_cluster.cc:1428] Setting key 35c8e3d8c852686b881701001c962524
I20260926 20:14:36.533016 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.524666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:45643 (local address 127.23.50.131:53875)
0926 20:14:36.525442 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:36.525498 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:36.526920 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:36.527502 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:36.527509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:36.527596 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:14:36.528103 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.528116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.528928 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.528931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.529914 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.529925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.530443 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.530447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.530612 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.531245 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 20:14:36.531247 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:14:36.531286 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 20:14:36.531351 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:14:36.531651 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:14:36.531663 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:14:36.531671 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 20:14:36.531824 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 20:14:36.532098 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20260926 20:14:36.534260 11521 heartbeater.cc:347] Connected to a master server at 127.23.50.190:45643
I20260926 20:14:36.534516 11521 heartbeater.cc:464] Registering TS with master...
I20260926 20:14:36.535051 11521 heartbeater.cc:511] Master 127.23.50.190:45643 requested a full tablet report, sending...
I20260926 20:14:36.535945 11071 ts_manager.cc:194] Registered new tserver with Master: eb7ee0a97a844a638923d5f9fffb91df (127.23.50.131:38337)
I20260926 20:14:36.536465 11071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.50.131:53875
I20260926 20:14:36.546617 23754 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 20:14:36.552512 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.548633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.23.50.190:45643)
0926 20:14:36.548838 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:36.548843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:36.548974 (+ 131us) server_negotiation.cc:407] Connection header received
0926 20:14:36.549124 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:36.549127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:36.549176 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:36.549261 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:36.549563 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.550154 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.551006 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.551218 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.551328 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.552101 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:36.552104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:36.552106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:36.552117 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:36.552134 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:36.552142 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:36.552218 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:36.552309 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 20:14:36.552371 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260926 20:14:36.555346 11071 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 20:14:36.558364 11071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 20:14:36.572921 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.567096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:38337 (local address 127.0.0.1:55460)
0926 20:14:36.568162 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:36.568176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:36.568260 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:36.569029 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:36.569033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:36.569051 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:36.569120 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.569125 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.570396 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.570399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.571564 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.571574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.572374 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.572377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.572482 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.572519 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 20:14:36.572638 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 20:14:36.572757 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":39,"thread_start_us":94,"threads_started":1}
I20260926 20:14:36.572927 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.567705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.23.50.131:38337)
0926 20:14:36.568488 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:36.568500 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:36.568524 (+ 24us) server_negotiation.cc:407] Connection header received
0926 20:14:36.568602 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:36.568606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:36.568782 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:36.568939 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:36.569238 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.570310 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.571684 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.572229 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.572357 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.572557 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:36.572667 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 20:14:36.572760 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":177,"threads_started":1}
I20260926 20:14:36.573328 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.566848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:45509 (local address 127.0.0.1:43646)
0926 20:14:36.567562 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:36.567613 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:36.567745 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:36.568485 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:36.568490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:36.568510 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:36.568651 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.568658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.570108 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.570111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.571531 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.571542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.572770 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.572774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.572871 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.572894 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 20:14:36.572963 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 20:14:36.573176 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":67,"thread_start_us":81,"threads_started":1}
I20260926 20:14:36.573833 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.567015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.23.50.129:45509)
0926 20:14:36.567368 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:36.567381 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:36.567898 (+ 517us) server_negotiation.cc:407] Connection header received
0926 20:14:36.567986 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:36.567991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:36.568238 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:36.568405 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:36.568785 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.569998 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.571672 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.572441 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.572535 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.572708 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:36.573580 (+ 872us) server_negotiation.cc:299] Negotiation successful
0926 20:14:36.573681 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":130,"threads_started":1}
I20260926 20:14:36.574119 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.567576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:42919 (local address 127.0.0.1:49142)
0926 20:14:36.568254 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:36.568269 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:36.568358 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:36.568790 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:36.568792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:36.568818 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:36.568908 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.568914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.569949 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.569953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.572415 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.572426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.573686 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.573690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.573783 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.573818 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 20:14:36.573918 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 20:14:36.573997 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":1167,"thread_start_us":80,"threads_started":1}
I20260926 20:14:36.574513 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.567738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.23.50.130:42919)
0926 20:14:36.568356 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:36.568369 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:36.568394 (+ 25us) server_negotiation.cc:407] Connection header received
0926 20:14:36.568475 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:36.568480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:36.568653 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:36.568793 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:36.568995 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.569795 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.572563 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.573340 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.573455 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.573628 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:36.574273 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 20:14:36.574368 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":123,"threads_started":1}
I20260926 20:14:36.576056 11455 tablet_service.cc:1467] Processing CreateTablet for tablet e51057389c0f474f8bb650be4c2bc242 (DEFAULT_TABLE table=test-table [id=c5430f272fb2462da3d1aaea69d31909]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:36.576511 11193 tablet_service.cc:1467] Processing CreateTablet for tablet e51057389c0f474f8bb650be4c2bc242 (DEFAULT_TABLE table=test-table [id=c5430f272fb2462da3d1aaea69d31909]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:36.577237 11455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e51057389c0f474f8bb650be4c2bc242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:36.577476 11193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e51057389c0f474f8bb650be4c2bc242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:36.577292 11324 tablet_service.cc:1467] Processing CreateTablet for tablet e51057389c0f474f8bb650be4c2bc242 (DEFAULT_TABLE table=test-table [id=c5430f272fb2462da3d1aaea69d31909]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 20:14:36.578285 11324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e51057389c0f474f8bb650be4c2bc242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:36.583621 11538 tablet_bootstrap.cc:492] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e: Bootstrap starting.
I20260926 20:14:36.585233 11539 tablet_bootstrap.cc:492] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df: Bootstrap starting.
I20260926 20:14:36.586019 11538 tablet_bootstrap.cc:654] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:36.586203 11540 tablet_bootstrap.cc:492] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f: Bootstrap starting.
I20260926 20:14:36.586809 11538 log.cc:824] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:36.587357 11539 tablet_bootstrap.cc:654] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:36.588222 11540 tablet_bootstrap.cc:654] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:36.588213 11539 log.cc:824] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:36.588973 11540 log.cc:824] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:36.589390 11538 tablet_bootstrap.cc:492] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e: No bootstrap required, opened a new log
I20260926 20:14:36.589598 11538 ts_tablet_manager.cc:1397] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 20:14:36.590742 11540 tablet_bootstrap.cc:492] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f: No bootstrap required, opened a new log
I20260926 20:14:36.590896 11540 ts_tablet_manager.cc:1397] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 20:14:36.592872 11538 raft_consensus.cc:348] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5713fb460bf44266a6e10d6f052dc38e" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45509 } } peers { permanent_uuid: "f9d0b57cecda4e2da3523a203df5159f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42919 } } peers { permanent_uuid: "eb7ee0a97a844a638923d5f9fffb91df" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38337 } }
I20260926 20:14:36.593119 11538 raft_consensus.cc:374] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:36.593178 11538 raft_consensus.cc:729] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5713fb460bf44266a6e10d6f052dc38e, State: Initialized, Role: FOLLOWER
I20260926 20:14:36.593626 11538 consensus_queue.cc:260] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5713fb460bf44266a6e10d6f052dc38e" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45509 } } peers { permanent_uuid: "f9d0b57cecda4e2da3523a203df5159f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42919 } } peers { permanent_uuid: "eb7ee0a97a844a638923d5f9fffb91df" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38337 } }
I20260926 20:14:36.594286 11259 heartbeater.cc:503] Master 127.23.50.190:45643 was elected leader, sending a full tablet report...
I20260926 20:14:36.594254 11540 raft_consensus.cc:348] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5713fb460bf44266a6e10d6f052dc38e" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45509 } } peers { permanent_uuid: "f9d0b57cecda4e2da3523a203df5159f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42919 } } peers { permanent_uuid: "eb7ee0a97a844a638923d5f9fffb91df" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38337 } }
I20260926 20:14:36.594497 11540 raft_consensus.cc:374] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:36.594563 11540 raft_consensus.cc:729] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d0b57cecda4e2da3523a203df5159f, State: Initialized, Role: FOLLOWER
I20260926 20:14:36.594666 11538 ts_tablet_manager.cc:1428] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260926 20:14:36.595018 11540 consensus_queue.cc:260] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5713fb460bf44266a6e10d6f052dc38e" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45509 } } peers { permanent_uuid: "f9d0b57cecda4e2da3523a203df5159f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42919 } } peers { permanent_uuid: "eb7ee0a97a844a638923d5f9fffb91df" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38337 } }
I20260926 20:14:36.595964 11540 ts_tablet_manager.cc:1428] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 20:14:36.596048 11539 tablet_bootstrap.cc:492] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df: No bootstrap required, opened a new log
I20260926 20:14:36.596220 11539 ts_tablet_manager.cc:1397] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20260926 20:14:36.596356 11390 heartbeater.cc:503] Master 127.23.50.190:45643 was elected leader, sending a full tablet report...
I20260926 20:14:36.599896 11539 raft_consensus.cc:348] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5713fb460bf44266a6e10d6f052dc38e" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45509 } } peers { permanent_uuid: "f9d0b57cecda4e2da3523a203df5159f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42919 } } peers { permanent_uuid: "eb7ee0a97a844a638923d5f9fffb91df" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38337 } }
I20260926 20:14:36.600121 11539 raft_consensus.cc:374] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:36.600196 11539 raft_consensus.cc:729] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7ee0a97a844a638923d5f9fffb91df, State: Initialized, Role: FOLLOWER
I20260926 20:14:36.600574 11539 consensus_queue.cc:260] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5713fb460bf44266a6e10d6f052dc38e" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45509 } } peers { permanent_uuid: "f9d0b57cecda4e2da3523a203df5159f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42919 } } peers { permanent_uuid: "eb7ee0a97a844a638923d5f9fffb91df" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38337 } }
I20260926 20:14:36.601253 11521 heartbeater.cc:503] Master 127.23.50.190:45643 was elected leader, sending a full tablet report...
I20260926 20:14:36.601346 11539 ts_tablet_manager.cc:1428] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 20:14:36.623049 11260 tablet.cc:2369] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:14:36.654524 11544 raft_consensus.cc:482] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 20:14:36.654750 11544 raft_consensus.cc:504] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5713fb460bf44266a6e10d6f052dc38e" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45509 } } peers { permanent_uuid: "f9d0b57cecda4e2da3523a203df5159f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42919 } } peers { permanent_uuid: "eb7ee0a97a844a638923d5f9fffb91df" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38337 } }
I20260926 20:14:36.656011 11544 leader_election.cc:288] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9d0b57cecda4e2da3523a203df5159f (127.23.50.130:42919), eb7ee0a97a844a638923d5f9fffb91df (127.23.50.131:38337)
I20260926 20:14:36.659801 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.656364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:53791 (local address 127.23.50.130:42919)
0926 20:14:36.656488 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:36.656492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:36.656638 (+ 146us) server_negotiation.cc:407] Connection header received
0926 20:14:36.656806 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:36.656809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:36.656861 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:36.656945 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:36.657225 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.657728 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.658835 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.659215 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.659371 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.659425 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:36.659607 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 20:14:36.659675 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:14:36.659825 11547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.656206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.130:42919 (local address 127.23.50.129:53791)
0926 20:14:36.656536 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:36.656555 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:36.656685 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:36.656959 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:36.656962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:36.656978 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:36.657107 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.657113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.657869 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.657872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.658704 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.658713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.659372 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.659376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.659474 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.659502 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 20:14:36.659591 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 20:14:36.659661 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260926 20:14:36.660024 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.656409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.50.129:44009 (local address 127.23.50.131:38337)
0926 20:14:36.656545 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:36.656549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:36.656850 (+ 301us) server_negotiation.cc:407] Connection header received
0926 20:14:36.657017 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:36.657025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:36.657066 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:36.657148 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 20:14:36.657437 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.657989 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.659027 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.659420 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.659545 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.659590 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:36.659789 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 20:14:36.659870 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:14:36.660332 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.656347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.131:38337 (local address 127.23.50.129:44009)
0926 20:14:36.656731 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:36.656744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:14:36.656892 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:14:36.657159 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:14:36.657162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:14:36.657178 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 20:14:36.657321 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.657327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.658099 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.658104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.658899 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:14:36.658908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.659542 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.659545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:14:36.659644 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.659671 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 20:14:36.659753 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 20:14:36.659841 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":25,"thread_start_us":124,"threads_started":1}
I20260926 20:14:36.660465 11344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e51057389c0f474f8bb650be4c2bc242" candidate_uuid: "5713fb460bf44266a6e10d6f052dc38e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d0b57cecda4e2da3523a203df5159f" is_pre_election: true
I20260926 20:14:36.660772 11344 raft_consensus.cc:2454] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5713fb460bf44266a6e10d6f052dc38e in term 0.
I20260926 20:14:36.660921 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e51057389c0f474f8bb650be4c2bc242" candidate_uuid: "5713fb460bf44266a6e10d6f052dc38e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7ee0a97a844a638923d5f9fffb91df" is_pre_election: true
I20260926 20:14:36.661206 11475 raft_consensus.cc:2454] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5713fb460bf44266a6e10d6f052dc38e in term 0.
I20260926 20:14:36.661265 11146 leader_election.cc:302] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5713fb460bf44266a6e10d6f052dc38e, f9d0b57cecda4e2da3523a203df5159f; no voters:
I20260926 20:14:36.661574 11544 raft_consensus.cc:2781] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 20:14:36.661685 11544 raft_consensus.cc:482] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 20:14:36.661737 11544 raft_consensus.cc:3037] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:36.662782 11544 raft_consensus.cc:504] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5713fb460bf44266a6e10d6f052dc38e" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45509 } } peers { permanent_uuid: "f9d0b57cecda4e2da3523a203df5159f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42919 } } peers { permanent_uuid: "eb7ee0a97a844a638923d5f9fffb91df" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38337 } }
I20260926 20:14:36.663117 11544 leader_election.cc:288] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [CANDIDATE]: Term 1 election: Requested vote from peers f9d0b57cecda4e2da3523a203df5159f (127.23.50.130:42919), eb7ee0a97a844a638923d5f9fffb91df (127.23.50.131:38337)
I20260926 20:14:36.663548 11475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e51057389c0f474f8bb650be4c2bc242" candidate_uuid: "5713fb460bf44266a6e10d6f052dc38e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7ee0a97a844a638923d5f9fffb91df"
I20260926 20:14:36.663565 11344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e51057389c0f474f8bb650be4c2bc242" candidate_uuid: "5713fb460bf44266a6e10d6f052dc38e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d0b57cecda4e2da3523a203df5159f"
I20260926 20:14:36.663682 11344 raft_consensus.cc:3037] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:36.663681 11475 raft_consensus.cc:3037] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:36.664845 11344 raft_consensus.cc:2454] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5713fb460bf44266a6e10d6f052dc38e in term 1.
I20260926 20:14:36.665022 11475 raft_consensus.cc:2454] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5713fb460bf44266a6e10d6f052dc38e in term 1.
I20260926 20:14:36.665172 11146 leader_election.cc:302] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5713fb460bf44266a6e10d6f052dc38e, f9d0b57cecda4e2da3523a203df5159f; no voters:
I20260926 20:14:36.665448 11544 raft_consensus.cc:2781] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:36.665671 11544 raft_consensus.cc:686] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [term 1 LEADER]: Becoming Leader. State: Replica: 5713fb460bf44266a6e10d6f052dc38e, State: Running, Role: LEADER
I20260926 20:14:36.666060 11544 consensus_queue.cc:237] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5713fb460bf44266a6e10d6f052dc38e" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45509 } } peers { permanent_uuid: "f9d0b57cecda4e2da3523a203df5159f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42919 } } peers { permanent_uuid: "eb7ee0a97a844a638923d5f9fffb91df" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38337 } }
I20260926 20:14:36.669261 11070 catalog_manager.cc:5495] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5713fb460bf44266a6e10d6f052dc38e (127.23.50.129). New cstate: current_term: 1 leader_uuid: "5713fb460bf44266a6e10d6f052dc38e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5713fb460bf44266a6e10d6f052dc38e" member_type: VOTER last_known_addr { host: "127.23.50.129" port: 45509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9d0b57cecda4e2da3523a203df5159f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb7ee0a97a844a638923d5f9fffb91df" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38337 } health_report { overall_health: UNKNOWN } } }
I20260926 20:14:36.696391 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.692794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43648 (local address 127.23.50.129:45509)
0926 20:14:36.692943 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:36.692947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:36.692964 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:14:36.693059 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:36.693061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:36.693115 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:36.693229 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:14:36.693489 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.694070 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.694934 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:36.695186 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:36.695306 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:36.695605 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:14:36.696015 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:14:36.696153 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:36.696196 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 20:14:36.696247 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:14:36.702234 11344 raft_consensus.cc:1261] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f [term 1 FOLLOWER]: Refusing update from remote peer 5713fb460bf44266a6e10d6f052dc38e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:14:36.702234 11475 raft_consensus.cc:1261] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df [term 1 FOLLOWER]: Refusing update from remote peer 5713fb460bf44266a6e10d6f052dc38e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:14:36.702946 11544 consensus_queue.cc:1035] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d0b57cecda4e2da3523a203df5159f" member_type: VOTER last_known_addr { host: "127.23.50.130" port: 42919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:14:36.703148 11549 consensus_queue.cc:1035] T e51057389c0f474f8bb650be4c2bc242 P 5713fb460bf44266a6e10d6f052dc38e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb7ee0a97a844a638923d5f9fffb91df" member_type: VOTER last_known_addr { host: "127.23.50.131" port: 38337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:14:36.713261 11553 mvcc.cc:205] Tried to move back new op lower bound from 7333698259765342208 to 7333698259626754048. Current Snapshot: MvccSnapshot[applied={T|T < 7333698259765342208}]
I20260926 20:14:36.719286 11071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:14:36.719473 11071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 20:14:36.721863 11071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5618a6900e9496289a1eb5060bee44d: Sending DeleteTablet for 3 replicas of tablet e51057389c0f474f8bb650be4c2bc242
I20260926 20:14:36.722751 11324 tablet_service.cc:1514] Processing DeleteTablet for tablet e51057389c0f474f8bb650be4c2bc242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:36 UTC) from {username='slave'} at 127.0.0.1:49142
I20260926 20:14:36.722752 11455 tablet_service.cc:1514] Processing DeleteTablet for tablet e51057389c0f474f8bb650be4c2bc242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:36 UTC) from {username='slave'} at 127.0.0.1:55460
I20260926 20:14:36.722782 11193 tablet_service.cc:1514] Processing DeleteTablet for tablet e51057389c0f474f8bb650be4c2bc242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 20:14:36 UTC) from {username='slave'} at 127.0.0.1:43646
I20260926 20:14:36.723258 11562 tablet_replica.cc:330] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df: stopping tablet replica
I20260926 20:14:36.723590 11562 raft_consensus.cc:2229] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:14:36.723930 11562 raft_consensus.cc:2258] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:36.723945 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 11131
I20260926 20:14:36.725046 11563 tablet_replica.cc:330] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f: stopping tablet replica
I20260926 20:14:36.725395 11563 raft_consensus.cc:2229] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 20:14:36.725703 11563 raft_consensus.cc:2258] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 20:14:36.726930 11562 ts_tablet_manager.cc:1905] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:14:36.727254 11563 ts_tablet_manager.cc:1905] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 20:14:36.729166 11562 ts_tablet_manager.cc:1918] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:14:36.729282 11562 log.cc:1196] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-2/wal/wals/e51057389c0f474f8bb650be4c2bc242
I20260926 20:14:36.729566 11562 ts_tablet_manager.cc:1939] T e51057389c0f474f8bb650be4c2bc242 P eb7ee0a97a844a638923d5f9fffb91df: Deleting consensus metadata
I20260926 20:14:36.730738 11563 ts_tablet_manager.cc:1918] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 20:14:36.730845 11563 log.cc:1196] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f: Deleting WAL directory at /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790453421225356-23754-0/minicluster-data/ts-1/wal/wals/e51057389c0f474f8bb650be4c2bc242
I20260926 20:14:36.730792 11056 catalog_manager.cc:4841] TS eb7ee0a97a844a638923d5f9fffb91df (127.23.50.131:38337): tablet e51057389c0f474f8bb650be4c2bc242 (table test-table [id=c5430f272fb2462da3d1aaea69d31909]) successfully deleted
I20260926 20:14:36.731153 11563 ts_tablet_manager.cc:1939] T e51057389c0f474f8bb650be4c2bc242 P f9d0b57cecda4e2da3523a203df5159f: Deleting consensus metadata
W20260926 20:14:36.732366 11055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.50.129:45509 (error 108)
I20260926 20:14:36.732836 11055 catalog_manager.cc:4841] TS f9d0b57cecda4e2da3523a203df5159f (127.23.50.130:42919): tablet e51057389c0f474f8bb650be4c2bc242 (table test-table [id=c5430f272fb2462da3d1aaea69d31909]) successfully deleted
I20260926 20:14:36.733378 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 11262
I20260926 20:14:36.733521 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:36.733156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.129:45509 (local address 127.0.0.1:43652)
0926 20:14:36.733333 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 20:14:36.733421 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.129:45509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 20:14:36.734103 11055 catalog_manager.cc:4584] TS 5713fb460bf44266a6e10d6f052dc38e (127.23.50.129:45509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e51057389c0f474f8bb650be4c2bc242: Network error: Client connection negotiation failed: client connection to 127.23.50.129:45509: connect: Connection refused (error 111)
I20260926 20:14:36.739336 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 11393
I20260926 20:14:36.744805 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 11040
2026-09-26T20:14:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1071 ms)
[----------] 6 tests from EncryptionPolicyTest (10829 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:36 dist-test-slave-7034 krb5kdc[11568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:36 dist-test-slave-7034 krb5kdc[11568](info): set up 2 sockets
Sep 26 20:14:36 dist-test-slave-7034 krb5kdc[11568](info): commencing operation
krb5kdc: starting...
W20260926 20:14:37.802924 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:37 dist-test-slave-7034 krb5kdc[11568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:14:37.929127 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:38.039354 11581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:38.039621 11581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:38.039662 11581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:38.043727 11581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:38.043793 11581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:38.043812 11581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:38.043829 11581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:38.043844 11581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:38.049044 11581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:38.050093 11581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:38.051085 11581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:14:38.059705 11581 server_base.cc:1034] running on GCE node
W20260926 20:14:38.059723 11589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:38.059710 11586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:38.059715 11587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:38.060593 11581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:38.061697 11581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:38.062855 11581 hybrid_clock.cc:648] HybridClock initialized: now 1790453678062822 us; error 47 us; skew 500 ppm
Sep 26 20:14:38 dist-test-slave-7034 krb5kdc[11568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:38.065604 11581 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:14:38.066557 11581 webserver.cc:460] Webserver started at http://127.0.0.1:35453/ using document root <none> and password file <none>
I20260926 20:14:38.067346 11581 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:38.067481 11581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:38.067735 11581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:38.069988 11581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "ade2a26290d64329828ef93c319bf16f"
format_stamp: "Formatted at 2026-09-26 20:14:38 on dist-test-slave-7034"
server_key: "60f8ede22734052d9b45261526f76be2"
server_key_iv: "81329fa1faa588272e3cc8496baa3108"
server_key_version: "encryptionkey@0"
I20260926 20:14:38.070721 11581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "ade2a26290d64329828ef93c319bf16f"
format_stamp: "Formatted at 2026-09-26 20:14:38 on dist-test-slave-7034"
server_key: "60f8ede22734052d9b45261526f76be2"
server_key_iv: "81329fa1faa588272e3cc8496baa3108"
server_key_version: "encryptionkey@0"
I20260926 20:14:38.074733 11581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 20:14:38.077317 11596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:38.078671 11581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:38.078826 11581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "ade2a26290d64329828ef93c319bf16f"
format_stamp: "Formatted at 2026-09-26 20:14:38 on dist-test-slave-7034"
server_key: "60f8ede22734052d9b45261526f76be2"
server_key_iv: "81329fa1faa588272e3cc8496baa3108"
server_key_version: "encryptionkey@0"
I20260926 20:14:38.078951 11581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:38.096397 11581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:38.099493 11581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:38.099709 11581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:38.107059 11581 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35383
I20260926 20:14:38.107064 11648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35383 every 8 connection(s)
I20260926 20:14:38.108047 11581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:38.110932 11649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:38.115932 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 11581
I20260926 20:14:38.116073 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:38.116539 23754 external_mini_cluster.cc:1428] Setting key 4ad2c7c80d1e2f07b16f0c3f0cdd41c8
I20260926 20:14:38.116982 11649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f: Bootstrap starting.
I20260926 20:14:38.119745 11649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:38.120585 11649 log.cc:824] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:38.123014 11649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f: No bootstrap required, opened a new log
Sep 26 20:14:38 dist-test-slave-7034 krb5kdc[11568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:14:38.127794 11649 raft_consensus.cc:348] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade2a26290d64329828ef93c319bf16f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35383 } }
I20260926 20:14:38.128031 11649 raft_consensus.cc:374] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:38.128744 11649 raft_consensus.cc:729] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade2a26290d64329828ef93c319bf16f, State: Initialized, Role: FOLLOWER
I20260926 20:14:38.129240 11649 consensus_queue.cc:260] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade2a26290d64329828ef93c319bf16f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35383 } }
I20260926 20:14:38.129417 11649 raft_consensus.cc:388] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:38.129515 11649 raft_consensus.cc:482] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:38.129655 11649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:38.130694 11649 raft_consensus.cc:504] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade2a26290d64329828ef93c319bf16f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35383 } }
I20260926 20:14:38.131074 11649 leader_election.cc:302] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ade2a26290d64329828ef93c319bf16f; no voters:
I20260926 20:14:38.131489 11649 leader_election.cc:288] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:38.131614 11654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:38.131807 11654 raft_consensus.cc:686] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [term 1 LEADER]: Becoming Leader. State: Replica: ade2a26290d64329828ef93c319bf16f, State: Running, Role: LEADER
I20260926 20:14:38.132145 11654 consensus_queue.cc:237] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade2a26290d64329828ef93c319bf16f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35383 } }
I20260926 20:14:38.132805 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:38.118147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35829 (local address 127.0.0.1:35383)
0926 20:14:38.118609 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:38.118624 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:38.118660 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:14:38.119305 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:38.119322 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:38.119663 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:38.120021 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:38.120877 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:38.122485 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:38.123434 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:38.124181 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:38.124352 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:38.127346 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:38.127351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:38.127364 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:38.127391 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:38.129395 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:38.130011 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:38.130020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:38.130031 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:38.130142 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:38.130497 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:38.130500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:38.130502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:38.130960 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:38.131177 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:38.131503 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 20:14:38.131947 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":143,"threads_started":1}
I20260926 20:14:38.132916 11649 sys_catalog.cc:564] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:38.133420 11654 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ade2a26290d64329828ef93c319bf16f. Latest consensus state: current_term: 1 leader_uuid: "ade2a26290d64329828ef93c319bf16f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade2a26290d64329828ef93c319bf16f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35383 } } }
I20260926 20:14:38.133543 11654 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:38.133800 11655 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ade2a26290d64329828ef93c319bf16f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade2a26290d64329828ef93c319bf16f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35383 } } }
I20260926 20:14:38.133927 11655 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:38.136343 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 20:14:38.138324 11669 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:14:38.138411 11669 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:14:38.138489 11658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:38.139472 11658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:38.145426 11658 catalog_manager.cc:1347] Generated new cluster ID: 5f6cfb96f13d4c14b1baae2fa8fb1e19
I20260926 20:14:38.145489 11658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:38.146979 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:38.138506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59119 (local address 127.0.0.1:35383)
0926 20:14:38.138620 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:38.138625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:38.138811 (+ 186us) server_negotiation.cc:407] Connection header received
0926 20:14:38.138924 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:38.138926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:38.138964 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:38.139034 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:38.139861 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:38.140383 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:38.141462 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:38.141694 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:38.141812 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:38.143210 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:38.143214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:38.143217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:38.143242 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:38.145092 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:38.145665 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:38.145668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:38.145669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:38.145737 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:38.146101 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:38.146104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:38.146106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:38.146279 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:38.146431 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:38.146700 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 20:14:38.146806 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 20:14:38.153980 11658 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:38.154820 11658 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:38.161105 11658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ade2a26290d64329828ef93c319bf16f: Generated new TSK 0
I20260926 20:14:38.161770 11658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:38.167629 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 11581
W20260926 20:14:38.173023 11566 connection.cc:535] client connection to 127.0.0.1:35383 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35383): Connection reset by peer (error 104)
2026-09-26T20:14:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:38 dist-test-slave-7034 krb5kdc[11680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:38 dist-test-slave-7034 krb5kdc[11680](info): set up 2 sockets
Sep 26 20:14:38 dist-test-slave-7034 krb5kdc[11680](info): commencing operation
krb5kdc: starting...
W20260926 20:14:39.214562 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:39 dist-test-slave-7034 krb5kdc[11680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:14:39.340708 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:39.449275 11693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:39.449582 11693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:39.449666 11693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:39.453642 11693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:39.453733 11693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:39.453773 11693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:39.453836 11693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:39.453891 11693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:39.459153 11693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43837
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:39.460242 11693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:39.461274 11693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:39.469925 11701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:39.470011 11699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:39.470044 11693 server_base.cc:1034] running on GCE node
W20260926 20:14:39.470259 11698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:39.470845 11693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:39.471973 11693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:39.473140 11693 hybrid_clock.cc:648] HybridClock initialized: now 1790453679473119 us; error 34 us; skew 500 ppm
Sep 26 20:14:39 dist-test-slave-7034 krb5kdc[11680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:39.475862 11693 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:14:39.476802 11693 webserver.cc:460] Webserver started at http://127.0.0.1:33489/ using document root <none> and password file <none>
I20260926 20:14:39.477361 11693 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:39.477453 11693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:39.477689 11693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:39.479837 11693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "199ce5f4f84a48d5a2656d1d7397ee5c"
format_stamp: "Formatted at 2026-09-26 20:14:39 on dist-test-slave-7034"
server_key: "4c79ad477d8e2c3c4866ef11360168f1"
server_key_iv: "3a67b2cbbf958c501ea383e63bff355a"
server_key_version: "encryptionkey@0"
I20260926 20:14:39.480393 11693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "199ce5f4f84a48d5a2656d1d7397ee5c"
format_stamp: "Formatted at 2026-09-26 20:14:39 on dist-test-slave-7034"
server_key: "4c79ad477d8e2c3c4866ef11360168f1"
server_key_iv: "3a67b2cbbf958c501ea383e63bff355a"
server_key_version: "encryptionkey@0"
I20260926 20:14:39.484042 11693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 20:14:39.486424 11708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:39.487653 11693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:39.487789 11693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "199ce5f4f84a48d5a2656d1d7397ee5c"
format_stamp: "Formatted at 2026-09-26 20:14:39 on dist-test-slave-7034"
server_key: "4c79ad477d8e2c3c4866ef11360168f1"
server_key_iv: "3a67b2cbbf958c501ea383e63bff355a"
server_key_version: "encryptionkey@0"
I20260926 20:14:39.487910 11693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:39.498843 11693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:39.501601 11693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:39.501806 11693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:39.509050 11693 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43837
I20260926 20:14:39.509074 11760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43837 every 8 connection(s)
I20260926 20:14:39.510139 11693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:39.512912 11761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:39.516947 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 11693
I20260926 20:14:39.517068 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:39.517392 23754 external_mini_cluster.cc:1428] Setting key 6653876d57a40616624cc53b1c2b42db
I20260926 20:14:39.518826 11761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c: Bootstrap starting.
I20260926 20:14:39.521591 11761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:39.522505 11761 log.cc:824] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:39.524585 11761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c: No bootstrap required, opened a new log
I20260926 20:14:39.527661 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:39.518833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57907 (local address 127.0.0.1:43837)
0926 20:14:39.519286 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:39.519304 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:39.519345 (+ 41us) server_negotiation.cc:407] Connection header received
0926 20:14:39.520205 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:39.520228 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:39.520664 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:39.521032 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:39.521482 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:39.522986 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:39.523930 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:39.524591 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:39.524751 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:39.525600 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:39.525605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:39.525621 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:39.526142 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:39.526207 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:39.526235 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:39.526376 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:39.526541 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 20:14:39.526677 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":125,"threads_started":1}
I20260926 20:14:39.528996 11761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199ce5f4f84a48d5a2656d1d7397ee5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } }
I20260926 20:14:39.529208 11761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:39.529274 11761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 199ce5f4f84a48d5a2656d1d7397ee5c, State: Initialized, Role: FOLLOWER
I20260926 20:14:39.529812 11761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199ce5f4f84a48d5a2656d1d7397ee5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } }
I20260926 20:14:39.530007 11761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:39.530084 11761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:39.530181 11761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:39.530683 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:39.531268 11761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199ce5f4f84a48d5a2656d1d7397ee5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } }
I20260926 20:14:39.531625 11761 leader_election.cc:302] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 199ce5f4f84a48d5a2656d1d7397ee5c; no voters:
I20260926 20:14:39.531992 11761 leader_election.cc:288] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:39.532092 11768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:39.532310 11768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [term 1 LEADER]: Becoming Leader. State: Replica: 199ce5f4f84a48d5a2656d1d7397ee5c, State: Running, Role: LEADER
I20260926 20:14:39.532706 11768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199ce5f4f84a48d5a2656d1d7397ee5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } }
I20260926 20:14:39.533370 11761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:39.533960 11772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 199ce5f4f84a48d5a2656d1d7397ee5c. Latest consensus state: current_term: 1 leader_uuid: "199ce5f4f84a48d5a2656d1d7397ee5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199ce5f4f84a48d5a2656d1d7397ee5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } } }
I20260926 20:14:39.534137 11772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:39.534466 11771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "199ce5f4f84a48d5a2656d1d7397ee5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199ce5f4f84a48d5a2656d1d7397ee5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43837 } } }
I20260926 20:14:39.534632 11771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:39.535012 11779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:39.538041 11779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:39.539323 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:39.532846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35199 (local address 127.0.0.1:43837)
0926 20:14:39.532948 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:39.532952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:39.534635 (+ 1683us) server_negotiation.cc:407] Connection header received
0926 20:14:39.534809 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:39.534812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:39.534862 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:39.534949 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:39.535480 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:39.536227 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:39.537391 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:39.537715 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:39.537835 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:39.538773 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:39.538777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:39.538779 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:39.538793 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:39.538828 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:39.538843 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:39.538908 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:39.539155 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 20:14:39.539196 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 20:14:39.543648 11779 catalog_manager.cc:1347] Generated new cluster ID: f8db82f9bad64a32bc7d720ef6d539e7
I20260926 20:14:39.543720 11779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:39.554622 11779 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:39.555585 11779 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:39.572614 11779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 199ce5f4f84a48d5a2656d1d7397ee5c: Generated new TSK 0
I20260926 20:14:39.573383 11779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:39.583559 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 11693
W20260926 20:14:39.588476 11678 connection.cc:535] client connection to 127.0.0.1:43837 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43837): Connection reset by peer (error 104)
2026-09-26T20:14:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:39 dist-test-slave-7034 krb5kdc[11792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:39 dist-test-slave-7034 krb5kdc[11792](info): set up 2 sockets
Sep 26 20:14:39 dist-test-slave-7034 krb5kdc[11792](info): commencing operation
krb5kdc: starting...
W20260926 20:14:40.614979 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:40 dist-test-slave-7034 krb5kdc[11792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:14:40.742230 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:40.852785 11805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:40.853097 11805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:40.853179 11805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:40.857314 11805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:40.857409 11805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:40.857450 11805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:40.857508 11805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:40.857547 11805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:40.862807 11805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:40.863879 11805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:40.864908 11805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:14:40.873624 11805 server_base.cc:1034] running on GCE node
W20260926 20:14:40.873570 11810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:40.873570 11813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:40.873570 11811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:40.874406 11805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:40.875505 11805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:40.876709 11805 hybrid_clock.cc:648] HybridClock initialized: now 1790453680876694 us; error 27 us; skew 500 ppm
Sep 26 20:14:40 dist-test-slave-7034 krb5kdc[11792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:40.879465 11805 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:14:40.880280 11805 webserver.cc:460] Webserver started at http://127.0.0.1:33311/ using document root <none> and password file <none>
I20260926 20:14:40.880787 11805 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:40.880843 11805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:40.881002 11805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:40.883081 11805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "985e52440d3f46f7a7a7e52af55daa81"
format_stamp: "Formatted at 2026-09-26 20:14:40 on dist-test-slave-7034"
server_key: "ba895314020d9fc3a64ab09e6fdf3996"
server_key_iv: "977a47da2b155886fd112b00acbfc095"
server_key_version: "encryptionkey@0"
I20260926 20:14:40.883602 11805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "985e52440d3f46f7a7a7e52af55daa81"
format_stamp: "Formatted at 2026-09-26 20:14:40 on dist-test-slave-7034"
server_key: "ba895314020d9fc3a64ab09e6fdf3996"
server_key_iv: "977a47da2b155886fd112b00acbfc095"
server_key_version: "encryptionkey@0"
I20260926 20:14:40.887197 11805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:14:40.889554 11820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:40.890921 11805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:40.891062 11805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "985e52440d3f46f7a7a7e52af55daa81"
format_stamp: "Formatted at 2026-09-26 20:14:40 on dist-test-slave-7034"
server_key: "ba895314020d9fc3a64ab09e6fdf3996"
server_key_iv: "977a47da2b155886fd112b00acbfc095"
server_key_version: "encryptionkey@0"
I20260926 20:14:40.891171 11805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:40.896031 11805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:40.898887 11805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:40.899086 11805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:40.905627 11805 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41653
I20260926 20:14:40.905687 11872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41653 every 8 connection(s)
I20260926 20:14:40.906634 11805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:40.908283 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 11805
I20260926 20:14:40.908452 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:40.908764 23754 external_mini_cluster.cc:1428] Setting key 90a3793e2827b5e98c609ab445f513bc
I20260926 20:14:40.909556 11873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:40.914376 11876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:40.910172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51381 (local address 127.0.0.1:41653)
0926 20:14:40.910587 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:40.910603 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:40.910638 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:14:40.911268 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:40.911282 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:40.911556 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:40.911841 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:40.912727 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:40.912731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:40.912744 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:40.912754 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:40.912807 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:40.912825 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:40.912937 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:40.913070 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 20:14:40.913330 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":138,"threads_started":1}
I20260926 20:14:40.916059 11873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81: Bootstrap starting.
I20260926 20:14:40.917299 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:40.918500 11873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:40.919363 11873 log.cc:824] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:40.921197 11876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:40.919419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57297 (local address 127.0.0.1:41653)
0926 20:14:40.919526 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:40.919529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:40.919768 (+ 239us) server_negotiation.cc:407] Connection header received
0926 20:14:40.920029 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:40.920033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:40.920064 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:40.920158 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:40.920833 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:40.920836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:40.920838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:40.920848 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:40.920871 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:40.920881 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:40.920948 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:40.921044 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 20:14:40.921073 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 20:14:40.921938 11873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81: No bootstrap required, opened a new log
I20260926 20:14:40.925369 11873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985e52440d3f46f7a7a7e52af55daa81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } }
I20260926 20:14:40.925585 11873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:40.925647 11873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 985e52440d3f46f7a7a7e52af55daa81, State: Initialized, Role: FOLLOWER
I20260926 20:14:40.926123 11873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985e52440d3f46f7a7a7e52af55daa81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } }
I20260926 20:14:40.926270 11873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:40.926343 11873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:40.926445 11873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:40.927351 11873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985e52440d3f46f7a7a7e52af55daa81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } }
I20260926 20:14:40.927712 11873 leader_election.cc:302] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 985e52440d3f46f7a7a7e52af55daa81; no voters:
I20260926 20:14:40.928085 11873 leader_election.cc:288] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:40.928192 11883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:40.928432 11883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [term 1 LEADER]: Becoming Leader. State: Replica: 985e52440d3f46f7a7a7e52af55daa81, State: Running, Role: LEADER
I20260926 20:14:40.928794 11883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985e52440d3f46f7a7a7e52af55daa81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } }
I20260926 20:14:40.929322 11873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:40.930418 11884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 985e52440d3f46f7a7a7e52af55daa81. Latest consensus state: current_term: 1 leader_uuid: "985e52440d3f46f7a7a7e52af55daa81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985e52440d3f46f7a7a7e52af55daa81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } } }
I20260926 20:14:40.930557 11884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:40.930950 11892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:40.930913 11885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "985e52440d3f46f7a7a7e52af55daa81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985e52440d3f46f7a7a7e52af55daa81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } } }
I20260926 20:14:40.931335 11885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:40.934219 11892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:40.939570 11892 catalog_manager.cc:1347] Generated new cluster ID: 518b30379681423abab39648c0b44be5
I20260926 20:14:40.939635 11892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:40.976461 11892 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:40.976868 11892 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:40.986599 11892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 985e52440d3f46f7a7a7e52af55daa81: Generated new TSK 0
I20260926 20:14:40.987367 11892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:40.992575 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 11805
2026-09-26T20:14:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:41 dist-test-slave-7034 krb5kdc[11904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:41 dist-test-slave-7034 krb5kdc[11904](info): set up 2 sockets
Sep 26 20:14:41 dist-test-slave-7034 krb5kdc[11904](info): commencing operation
krb5kdc: starting...
W20260926 20:14:42.034662 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:42 dist-test-slave-7034 krb5kdc[11904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:14:42.160852 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:42.270747 11917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:42.271044 11917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:42.271126 11917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:42.275115 11917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:42.275215 11917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:42.275256 11917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:42.275321 11917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:42.275354 11917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:42.280630 11917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:42.281715 11917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:42.282734 11917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:42.291270 11922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:42.291410 11923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:42.291445 11917 server_base.cc:1034] running on GCE node
W20260926 20:14:42.291325 11925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:42.292130 11917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:42.293210 11917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:42.294395 11917 hybrid_clock.cc:648] HybridClock initialized: now 1790453682294377 us; error 31 us; skew 500 ppm
Sep 26 20:14:42 dist-test-slave-7034 krb5kdc[11904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:42.297099 11917 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:14:42.298069 11917 webserver.cc:460] Webserver started at http://127.0.0.1:34893/ using document root <none> and password file <none>
I20260926 20:14:42.298615 11917 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:42.298707 11917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:42.298944 11917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:42.300980 11917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "c73596c864664578be887db5bec29d48"
format_stamp: "Formatted at 2026-09-26 20:14:42 on dist-test-slave-7034"
server_key: "316ecfc56c8bf4e11da98b16c8bdba97"
server_key_iv: "d0312e46f251c2d55eae9de3a7831b98"
server_key_version: "encryptionkey@0"
I20260926 20:14:42.301560 11917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "c73596c864664578be887db5bec29d48"
format_stamp: "Formatted at 2026-09-26 20:14:42 on dist-test-slave-7034"
server_key: "316ecfc56c8bf4e11da98b16c8bdba97"
server_key_iv: "d0312e46f251c2d55eae9de3a7831b98"
server_key_version: "encryptionkey@0"
I20260926 20:14:42.305253 11917 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 20:14:42.307730 11932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:42.309003 11917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:14:42.309160 11917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "c73596c864664578be887db5bec29d48"
format_stamp: "Formatted at 2026-09-26 20:14:42 on dist-test-slave-7034"
server_key: "316ecfc56c8bf4e11da98b16c8bdba97"
server_key_iv: "d0312e46f251c2d55eae9de3a7831b98"
server_key_version: "encryptionkey@0"
I20260926 20:14:42.309279 11917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:42.341090 11917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:42.344231 11917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:42.344447 11917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:42.351724 11917 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43763
I20260926 20:14:42.351780 11984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43763 every 8 connection(s)
I20260926 20:14:42.352728 11917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:42.355581 11985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:42.357311 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 11917
I20260926 20:14:42.357451 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:42.357755 23754 external_mini_cluster.cc:1428] Setting key 1b44e5ef46a1decb3783a13ce29790bd
I20260926 20:14:42.361763 11985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48: Bootstrap starting.
I20260926 20:14:42.363957 11985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:42.364672 11985 log.cc:824] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48: Log is configured to *not* fsync() on all Append() calls
Sep 26 20:14:42 dist-test-slave-7034 krb5kdc[11904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:14:42.366794 11985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48: No bootstrap required, opened a new log
I20260926 20:14:42.369822 11985 raft_consensus.cc:348] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73596c864664578be887db5bec29d48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43763 } }
I20260926 20:14:42.370085 11985 raft_consensus.cc:374] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:42.370227 11985 raft_consensus.cc:729] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73596c864664578be887db5bec29d48, State: Initialized, Role: FOLLOWER
I20260926 20:14:42.370729 11985 consensus_queue.cc:260] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73596c864664578be887db5bec29d48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43763 } }
I20260926 20:14:42.370898 11985 raft_consensus.cc:388] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:42.370978 11985 raft_consensus.cc:482] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:42.371080 11985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:42.372048 11985 raft_consensus.cc:504] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73596c864664578be887db5bec29d48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43763 } }
I20260926 20:14:42.372401 11985 leader_election.cc:302] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c73596c864664578be887db5bec29d48; no voters:
I20260926 20:14:42.372496 11988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:42.359189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60343 (local address 127.0.0.1:43763)
0926 20:14:42.359622 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:42.359636 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:42.359671 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:14:42.360287 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:42.360314 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:42.360647 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:42.360981 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:42.361787 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:42.362810 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:42.363675 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:42.364201 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:42.364375 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:42.367164 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:42.367169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:42.367183 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:42.367209 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:42.369379 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:42.369926 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:42.369930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:42.369935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:42.370042 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:42.370399 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:42.370403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:42.370404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:42.370898 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:42.371118 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:42.371402 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 20:14:42.371692 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":127,"threads_started":1}
I20260926 20:14:42.372778 11985 leader_election.cc:288] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:42.372853 11990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:42.373076 11990 raft_consensus.cc:686] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [term 1 LEADER]: Becoming Leader. State: Replica: c73596c864664578be887db5bec29d48, State: Running, Role: LEADER
I20260926 20:14:42.373420 11990 consensus_queue.cc:237] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73596c864664578be887db5bec29d48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43763 } }
I20260926 20:14:42.374074 11985 sys_catalog.cc:564] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:42.374626 11990 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c73596c864664578be887db5bec29d48. Latest consensus state: current_term: 1 leader_uuid: "c73596c864664578be887db5bec29d48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73596c864664578be887db5bec29d48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43763 } } }
I20260926 20:14:42.375092 11990 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:42.375056 11991 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c73596c864664578be887db5bec29d48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73596c864664578be887db5bec29d48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43763 } } }
I20260926 20:14:42.375589 11991 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:42.375735 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:42.375919 11999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:42.379051 11999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:42.384428 11999 catalog_manager.cc:1347] Generated new cluster ID: 0e321ac1dd4a47e6bf3d2b4b1554fc5c
I20260926 20:14:42.384532 11999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:42.389300 11988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:42.378293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36969 (local address 127.0.0.1:43763)
0926 20:14:42.378400 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:42.378405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:42.380734 (+ 2329us) server_negotiation.cc:407] Connection header received
0926 20:14:42.380891 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:42.380894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:42.380938 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:42.381026 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:42.381684 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:42.382772 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:42.384056 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:42.384388 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:42.384487 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:42.386121 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:42.386125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:42.386128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:42.386154 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:42.387617 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:42.388051 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:42.388054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:42.388056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:42.388175 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:42.388466 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:42.388470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:42.388472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:42.388647 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:42.388783 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:42.389021 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 20:14:42.389133 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:14:42.402812 11999 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:42.403807 11999 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:42.410181 11999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c73596c864664578be887db5bec29d48: Generated new TSK 0
I20260926 20:14:42.410897 11999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:42.416469 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 11917
2026-09-26T20:14:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:42 dist-test-slave-7034 krb5kdc[12016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:42 dist-test-slave-7034 krb5kdc[12016](info): set up 2 sockets
Sep 26 20:14:42 dist-test-slave-7034 krb5kdc[12016](info): commencing operation
krb5kdc: starting...
W20260926 20:14:43.470512 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:43 dist-test-slave-7034 krb5kdc[12016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:14:43.595214 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:43.703454 12029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:43.703714 12029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:43.703755 12029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:43.707697 12029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:43.707756 12029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:43.707775 12029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:43.707792 12029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:43.707808 12029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:43.713028 12029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:43.714054 12029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:43.715006 12029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:43.724036 12034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:43.724054 12037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:43.724143 12029 server_base.cc:1034] running on GCE node
W20260926 20:14:43.724375 12035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:43.724977 12029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:43.726212 12029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:43.727402 12029 hybrid_clock.cc:648] HybridClock initialized: now 1790453683727356 us; error 65 us; skew 500 ppm
Sep 26 20:14:43 dist-test-slave-7034 krb5kdc[12016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:43.730952 12029 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:14:43.732069 12029 webserver.cc:460] Webserver started at http://127.0.0.1:46025/ using document root <none> and password file <none>
I20260926 20:14:43.732713 12029 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:43.732786 12029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:43.733024 12029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:43.735384 12029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "9c428eaa80db43e8a068dc1dd4cf3095"
format_stamp: "Formatted at 2026-09-26 20:14:43 on dist-test-slave-7034"
server_key: "500bb5a423dddbc1dd4a83411047bb2e"
server_key_iv: "6c9d10b3ce02e9bfb857973aa0d69db3"
server_key_version: "encryptionkey@0"
I20260926 20:14:43.736012 12029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "9c428eaa80db43e8a068dc1dd4cf3095"
format_stamp: "Formatted at 2026-09-26 20:14:43 on dist-test-slave-7034"
server_key: "500bb5a423dddbc1dd4a83411047bb2e"
server_key_iv: "6c9d10b3ce02e9bfb857973aa0d69db3"
server_key_version: "encryptionkey@0"
I20260926 20:14:43.740253 12029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:14:43.742996 12044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:43.744359 12029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 20:14:43.744488 12029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "9c428eaa80db43e8a068dc1dd4cf3095"
format_stamp: "Formatted at 2026-09-26 20:14:43 on dist-test-slave-7034"
server_key: "500bb5a423dddbc1dd4a83411047bb2e"
server_key_iv: "6c9d10b3ce02e9bfb857973aa0d69db3"
server_key_version: "encryptionkey@0"
I20260926 20:14:43.744602 12029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:43.780537 12029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:43.783716 12029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:43.783926 12029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:43.791095 12029 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38411
I20260926 20:14:43.791116 12096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38411 every 8 connection(s)
I20260926 20:14:43.792086 12029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:43.794976 12097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:43.800621 12097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095: Bootstrap starting.
I20260926 20:14:43.801476 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 12029
I20260926 20:14:43.801610 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:43.801961 23754 external_mini_cluster.cc:1428] Setting key 7a219f8e09f7f1ebf760a96b3a6d9104
I20260926 20:14:43.803613 12097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:43.804543 12097 log.cc:824] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:43.806859 12097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095: No bootstrap required, opened a new log
I20260926 20:14:43.811694 12097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c428eaa80db43e8a068dc1dd4cf3095" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260926 20:14:43.811983 12097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:43.812054 12097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c428eaa80db43e8a068dc1dd4cf3095, State: Initialized, Role: FOLLOWER
I20260926 20:14:43.812486 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:43.803092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59595 (local address 127.0.0.1:38411)
0926 20:14:43.803597 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:43.803620 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:43.803667 (+ 47us) server_negotiation.cc:407] Connection header received
0926 20:14:43.804606 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:43.804633 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:43.805083 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:43.805454 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:43.805922 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:43.807504 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:43.808477 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:43.809248 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:43.809414 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:43.810350 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:43.810359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:43.810377 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:43.810930 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:43.811001 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:43.811032 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:43.811145 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:43.811255 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 20:14:43.811473 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":141,"threads_started":1}
I20260926 20:14:43.812570 12097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c428eaa80db43e8a068dc1dd4cf3095" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260926 20:14:43.812780 12097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:43.812863 12097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:43.812974 12097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:43.814159 12097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c428eaa80db43e8a068dc1dd4cf3095" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260926 20:14:43.814571 12097 leader_election.cc:302] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c428eaa80db43e8a068dc1dd4cf3095; no voters:
I20260926 20:14:43.815006 12097 leader_election.cc:288] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:43.815075 12102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:43.815339 12102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [term 1 LEADER]: Becoming Leader. State: Replica: 9c428eaa80db43e8a068dc1dd4cf3095, State: Running, Role: LEADER
I20260926 20:14:43.815667 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:43.815708 12102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c428eaa80db43e8a068dc1dd4cf3095" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260926 20:14:43.816435 12097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:43.817060 12104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c428eaa80db43e8a068dc1dd4cf3095. Latest consensus state: current_term: 1 leader_uuid: "9c428eaa80db43e8a068dc1dd4cf3095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c428eaa80db43e8a068dc1dd4cf3095" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } } }
I20260926 20:14:43.817211 12104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:43.817597 12113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:43.819399 12103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c428eaa80db43e8a068dc1dd4cf3095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c428eaa80db43e8a068dc1dd4cf3095" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } } }
I20260926 20:14:43.819509 12103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:43.820746 12113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:43.823616 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:43.818046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45985 (local address 127.0.0.1:38411)
0926 20:14:43.818172 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:43.818180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:43.819077 (+ 897us) server_negotiation.cc:407] Connection header received
0926 20:14:43.819211 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:43.819214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:43.819283 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:43.819370 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:43.819681 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:43.820557 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:43.821553 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:43.821966 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:43.822104 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:43.823149 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:43.823153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:43.823156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:43.823175 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:43.823202 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:43.823215 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:43.823308 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:43.823414 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 20:14:43.823469 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:14:43.827131 12113 catalog_manager.cc:1347] Generated new cluster ID: d249b11142ff44629aebc988b608bf9a
I20260926 20:14:43.827203 12113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:43.843312 12113 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:43.844502 12113 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:43.849026 12113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c428eaa80db43e8a068dc1dd4cf3095: Generated new TSK 0
I20260926 20:14:43.849819 12113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:43.854488 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 12029
2026-09-26T20:14:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:43 dist-test-slave-7034 krb5kdc[12128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:43 dist-test-slave-7034 krb5kdc[12128](info): set up 2 sockets
Sep 26 20:14:43 dist-test-slave-7034 krb5kdc[12128](info): commencing operation
krb5kdc: starting...
W20260926 20:14:44.886575 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:44 dist-test-slave-7034 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:14:45.008764 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:45.120851 12141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:45.121163 12141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:45.121243 12141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:45.125325 12141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:45.125406 12141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:45.125442 12141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:45.125495 12141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:45.125535 12141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:45.130836 12141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:45.131913 12141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:45.132915 12141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:45.141242 12149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:45.141213 12147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:45.141420 12141 server_base.cc:1034] running on GCE node
W20260926 20:14:45.141242 12146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:45.142216 12141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:45.143262 12141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:45.144415 12141 hybrid_clock.cc:648] HybridClock initialized: now 1790453685144402 us; error 27 us; skew 500 ppm
Sep 26 20:14:45 dist-test-slave-7034 krb5kdc[12128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:45.147226 12141 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:14:45.148154 12141 webserver.cc:460] Webserver started at http://127.0.0.1:36103/ using document root <none> and password file <none>
I20260926 20:14:45.148721 12141 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:45.148783 12141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:45.148998 12141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:45.151103 12141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "af64d1e35fc04a70b4c4e05cce712902"
format_stamp: "Formatted at 2026-09-26 20:14:45 on dist-test-slave-7034"
server_key: "d692ccfcf82397c9146155b549d35303"
server_key_iv: "befc8621d79026ea4c7bd141987ecc72"
server_key_version: "encryptionkey@0"
I20260926 20:14:45.151662 12141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "af64d1e35fc04a70b4c4e05cce712902"
format_stamp: "Formatted at 2026-09-26 20:14:45 on dist-test-slave-7034"
server_key: "d692ccfcf82397c9146155b549d35303"
server_key_iv: "befc8621d79026ea4c7bd141987ecc72"
server_key_version: "encryptionkey@0"
I20260926 20:14:45.155303 12141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:14:45.157617 12156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:45.158928 12141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:45.159061 12141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "af64d1e35fc04a70b4c4e05cce712902"
format_stamp: "Formatted at 2026-09-26 20:14:45 on dist-test-slave-7034"
server_key: "d692ccfcf82397c9146155b549d35303"
server_key_iv: "befc8621d79026ea4c7bd141987ecc72"
server_key_version: "encryptionkey@0"
I20260926 20:14:45.159168 12141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:45.164230 12141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:45.167361 12141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:45.167507 12141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:45.173924 12141 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45675
I20260926 20:14:45.174235 12208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45675 every 8 connection(s)
I20260926 20:14:45.174887 12141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:45.177608 12209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:45.183097 12209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902: Bootstrap starting.
I20260926 20:14:45.184770 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 12141
I20260926 20:14:45.184921 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:45.185197 23754 external_mini_cluster.cc:1428] Setting key fcb8e6d6d209bde33e4b7f9f63f97929
I20260926 20:14:45.185757 12209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:45.186637 12209 log.cc:824] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:45.188536 12209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902: No bootstrap required, opened a new log
I20260926 20:14:45.190228 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:45.186435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47119 (local address 127.0.0.1:45675)
0926 20:14:45.186915 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:45.186929 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:45.186964 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:14:45.187600 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:45.187615 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:45.187888 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:45.188273 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:45.189014 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:45.189024 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:45.189036 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:45.189046 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:45.189095 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:45.189114 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:45.189236 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:45.189315 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 20:14:45.189470 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":132,"threads_started":1}
I20260926 20:14:45.192524 12209 raft_consensus.cc:348] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af64d1e35fc04a70b4c4e05cce712902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260926 20:14:45.192808 12209 raft_consensus.cc:374] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:45.192867 12209 raft_consensus.cc:729] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af64d1e35fc04a70b4c4e05cce712902, State: Initialized, Role: FOLLOWER
I20260926 20:14:45.193186 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:45.193344 12209 consensus_queue.cc:260] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af64d1e35fc04a70b4c4e05cce712902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260926 20:14:45.193513 12209 raft_consensus.cc:388] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:45.193598 12209 raft_consensus.cc:482] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:45.193694 12209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:45.194783 12209 raft_consensus.cc:504] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af64d1e35fc04a70b4c4e05cce712902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260926 20:14:45.195135 12209 leader_election.cc:302] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af64d1e35fc04a70b4c4e05cce712902; no voters:
I20260926 20:14:45.195448 12209 leader_election.cc:288] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:45.195657 12218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:45.195921 12218 raft_consensus.cc:686] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [term 1 LEADER]: Becoming Leader. State: Replica: af64d1e35fc04a70b4c4e05cce712902, State: Running, Role: LEADER
I20260926 20:14:45.196225 12218 consensus_queue.cc:237] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af64d1e35fc04a70b4c4e05cce712902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } }
I20260926 20:14:45.196738 12209 sys_catalog.cc:564] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:45.197952 12219 sys_catalog.cc:455] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af64d1e35fc04a70b4c4e05cce712902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af64d1e35fc04a70b4c4e05cce712902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } } }
I20260926 20:14:45.197952 12220 sys_catalog.cc:455] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af64d1e35fc04a70b4c4e05cce712902. Latest consensus state: current_term: 1 leader_uuid: "af64d1e35fc04a70b4c4e05cce712902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af64d1e35fc04a70b4c4e05cce712902" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45675 } } }
I20260926 20:14:45.198132 12219 sys_catalog.cc:458] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:45.198146 12220 sys_catalog.cc:458] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:45.201145 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:45.199271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33035 (local address 127.0.0.1:45675)
0926 20:14:45.199387 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:45.199391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:45.199692 (+ 301us) server_negotiation.cc:407] Connection header received
0926 20:14:45.199857 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:45.199860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:45.199896 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:45.199978 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:45.200747 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:45.200752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:45.200754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:45.200766 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:45.200799 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:45.200812 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:45.200902 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:45.200990 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 20:14:45.201027 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 20:14:45.201890 12233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:14:45.201970 12233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:14:45.202116 12228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:45.202811 12228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:45.208842 12228 catalog_manager.cc:1347] Generated new cluster ID: 910a87bb07ef4f78967c3e8f858c69fe
I20260926 20:14:45.208914 12228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:45.218633 12228 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:45.218988 12228 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:45.225262 12228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af64d1e35fc04a70b4c4e05cce712902: Generated new TSK 0
I20260926 20:14:45.226125 12228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:45.235499 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 12141
2026-09-26T20:14:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:45 dist-test-slave-7034 krb5kdc[12240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:45 dist-test-slave-7034 krb5kdc[12240](info): set up 2 sockets
Sep 26 20:14:45 dist-test-slave-7034 krb5kdc[12240](info): commencing operation
krb5kdc: starting...
W20260926 20:14:46.270624 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:46 dist-test-slave-7034 krb5kdc[12240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:14:46.398010 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:46.507766 12253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:46.508026 12253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:46.508122 12253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:46.512095 12253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:46.512148 12253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:46.512167 12253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:46.512214 12253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:46.512248 12253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:46.518537 12253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:46.519605 12253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:46.520622 12253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:46.528883 12259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:46.528855 12258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:46.528864 12261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:46.529152 12253 server_base.cc:1034] running on GCE node
I20260926 20:14:46.529770 12253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:46.530956 12253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:46.532106 12253 hybrid_clock.cc:648] HybridClock initialized: now 1790453686532057 us; error 67 us; skew 500 ppm
Sep 26 20:14:46 dist-test-slave-7034 krb5kdc[12240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:46.534966 12253 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:14:46.535876 12253 webserver.cc:460] Webserver started at http://127.0.0.1:44493/ using document root <none> and password file <none>
I20260926 20:14:46.536429 12253 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:46.536514 12253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:46.536737 12253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:46.538807 12253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "3591963e737e48a586f0e88be0e89c1d"
format_stamp: "Formatted at 2026-09-26 20:14:46 on dist-test-slave-7034"
server_key: "1d14daaa77770629b9ff864e4ea238e6"
server_key_iv: "237906a7caf89645b605dadb3b715a60"
server_key_version: "encryptionkey@0"
I20260926 20:14:46.539371 12253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "3591963e737e48a586f0e88be0e89c1d"
format_stamp: "Formatted at 2026-09-26 20:14:46 on dist-test-slave-7034"
server_key: "1d14daaa77770629b9ff864e4ea238e6"
server_key_iv: "237906a7caf89645b605dadb3b715a60"
server_key_version: "encryptionkey@0"
I20260926 20:14:46.543032 12253 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:14:46.545490 12268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:46.546768 12253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 20:14:46.546906 12253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "3591963e737e48a586f0e88be0e89c1d"
format_stamp: "Formatted at 2026-09-26 20:14:46 on dist-test-slave-7034"
server_key: "1d14daaa77770629b9ff864e4ea238e6"
server_key_iv: "237906a7caf89645b605dadb3b715a60"
server_key_version: "encryptionkey@0"
I20260926 20:14:46.547022 12253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:46.567015 12253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:46.570178 12253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:46.570376 12253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:46.577526 12253 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36699
I20260926 20:14:46.577548 12320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36699 every 8 connection(s)
I20260926 20:14:46.578534 12253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:46.581215 12321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:46.584002 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 12253
I20260926 20:14:46.584146 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:46.584455 23754 external_mini_cluster.cc:1428] Setting key 373ef0805d5d2c0393d5ac64648812cc
I20260926 20:14:46.587332 12321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d: Bootstrap starting.
I20260926 20:14:46.589493 12321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:46.590291 12321 log.cc:824] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:46.592191 12321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d: No bootstrap required, opened a new log
Sep 26 20:14:46 dist-test-slave-7034 krb5kdc[12240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:14:46.596520 12321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3591963e737e48a586f0e88be0e89c1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36699 } }
I20260926 20:14:46.596808 12321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:46.596871 12321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3591963e737e48a586f0e88be0e89c1d, State: Initialized, Role: FOLLOWER
I20260926 20:14:46.597316 12321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3591963e737e48a586f0e88be0e89c1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36699 } }
I20260926 20:14:46.597436 12321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:46.597518 12321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:46.597635 12321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:46.598667 12321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3591963e737e48a586f0e88be0e89c1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36699 } }
I20260926 20:14:46.599001 12321 leader_election.cc:302] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3591963e737e48a586f0e88be0e89c1d; no voters:
I20260926 20:14:46.599156 12324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:46.585895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:54421 (local address 127.0.0.1:36699)
0926 20:14:46.586325 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:46.586341 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:46.586376 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:14:46.587014 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:46.587032 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:46.587381 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:46.587755 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:46.588599 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:46.589639 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:46.590614 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:46.591097 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:46.591266 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:46.594091 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:46.594096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:46.594110 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:46.594138 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:46.595867 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:46.596427 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:46.596432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:46.596436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:46.596573 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:46.596980 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:46.596987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:46.596990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:46.597483 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:46.597729 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:46.598064 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 20:14:46.598303 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":137,"threads_started":1}
I20260926 20:14:46.599315 12321 leader_election.cc:288] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:46.599452 12326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:46.599650 12326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [term 1 LEADER]: Becoming Leader. State: Replica: 3591963e737e48a586f0e88be0e89c1d, State: Running, Role: LEADER
I20260926 20:14:46.600018 12326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3591963e737e48a586f0e88be0e89c1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36699 } }
I20260926 20:14:46.600608 12321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:46.601914 12327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3591963e737e48a586f0e88be0e89c1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3591963e737e48a586f0e88be0e89c1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36699 } } }
I20260926 20:14:46.602031 12327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:46.602344 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:46.602421 12328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3591963e737e48a586f0e88be0e89c1d. Latest consensus state: current_term: 1 leader_uuid: "3591963e737e48a586f0e88be0e89c1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3591963e737e48a586f0e88be0e89c1d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36699 } } }
I20260926 20:14:46.602519 12328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:46.602844 12335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:46.606174 12335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:46.611829 12335 catalog_manager.cc:1347] Generated new cluster ID: 850b62fef7fc4413934be6075cce6d4c
I20260926 20:14:46.611887 12335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:46.613687 12324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:46.604643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:33665 (local address 127.0.0.1:36699)
0926 20:14:46.604764 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:46.604769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:46.605001 (+ 232us) server_negotiation.cc:407] Connection header received
0926 20:14:46.605141 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:46.605144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:46.605192 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:46.605274 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:46.606093 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:46.606859 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:46.608014 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:46.608357 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:46.608466 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:46.609773 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:46.609779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:46.609784 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:46.609819 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:46.611801 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:46.612388 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:46.612392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:46.612394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:46.612483 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:46.612843 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:46.612846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:46.612847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:46.613029 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:46.613175 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:46.613449 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 20:14:46.613544 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 20:14:46.620121 12335 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:46.621248 12335 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:46.627215 12335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3591963e737e48a586f0e88be0e89c1d: Generated new TSK 0
I20260926 20:14:46.627995 12335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:46.632776 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 12253
W20260926 20:14:46.638257 12238 connection.cc:535] client connection to 127.0.0.1:36699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36699): Connection reset by peer (error 104)
2026-09-26T20:14:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:46 dist-test-slave-7034 krb5kdc[12352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:46 dist-test-slave-7034 krb5kdc[12352](info): set up 2 sockets
Sep 26 20:14:46 dist-test-slave-7034 krb5kdc[12352](info): commencing operation
krb5kdc: starting...
W20260926 20:14:47.666617 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:47 dist-test-slave-7034 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:14:47.793008 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:47.901347 12365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:47.901641 12365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:47.901715 12365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:47.905741 12365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:47.905844 12365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:47.905897 12365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:47.905936 12365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:47.905966 12365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:47.911316 12365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:47.912417 12365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:47.913457 12365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:14:47.922163 12365 server_base.cc:1034] running on GCE node
W20260926 20:14:47.922103 12370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:47.922103 12373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:47.922143 12371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:47.922988 12365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:47.924074 12365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:47.925258 12365 hybrid_clock.cc:648] HybridClock initialized: now 1790453687925235 us; error 41 us; skew 500 ppm
Sep 26 20:14:47 dist-test-slave-7034 krb5kdc[12352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:47.928128 12365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:14:47.929095 12365 webserver.cc:460] Webserver started at http://127.0.0.1:35557/ using document root <none> and password file <none>
I20260926 20:14:47.929628 12365 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:47.929690 12365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:47.929929 12365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:47.931994 12365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "a77ac040e6d54b6ab4e41315bd9d0537"
format_stamp: "Formatted at 2026-09-26 20:14:47 on dist-test-slave-7034"
server_key: "6ca7e7ea3bd202fa58157cebbc35171c"
server_key_iv: "ae5d444e428e7c71653bfdb1f708d80c"
server_key_version: "encryptionkey@0"
I20260926 20:14:47.932503 12365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "a77ac040e6d54b6ab4e41315bd9d0537"
format_stamp: "Formatted at 2026-09-26 20:14:47 on dist-test-slave-7034"
server_key: "6ca7e7ea3bd202fa58157cebbc35171c"
server_key_iv: "ae5d444e428e7c71653bfdb1f708d80c"
server_key_version: "encryptionkey@0"
I20260926 20:14:47.936213 12365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 20:14:47.938692 12380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:47.939994 12365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 20:14:47.940127 12365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "a77ac040e6d54b6ab4e41315bd9d0537"
format_stamp: "Formatted at 2026-09-26 20:14:47 on dist-test-slave-7034"
server_key: "6ca7e7ea3bd202fa58157cebbc35171c"
server_key_iv: "ae5d444e428e7c71653bfdb1f708d80c"
server_key_version: "encryptionkey@0"
I20260926 20:14:47.940234 12365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:47.957993 12365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:47.960881 12365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:47.961083 12365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:47.968308 12365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44907
I20260926 20:14:47.968336 12432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44907 every 8 connection(s)
I20260926 20:14:47.969250 12365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:47.971977 12433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:47.977511 12433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537: Bootstrap starting.
I20260926 20:14:47.979341 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 12365
I20260926 20:14:47.979481 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:47.979771 23754 external_mini_cluster.cc:1428] Setting key 468dcdc011f828d0723f56c1961f3d36
I20260926 20:14:47.980384 12433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:47.981323 12433 log.cc:824] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:47.983234 12433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537: No bootstrap required, opened a new log
I20260926 20:14:47.986739 12433 raft_consensus.cc:348] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77ac040e6d54b6ab4e41315bd9d0537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44907 } }
I20260926 20:14:47.986994 12433 raft_consensus.cc:374] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:47.987071 12433 raft_consensus.cc:729] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77ac040e6d54b6ab4e41315bd9d0537, State: Initialized, Role: FOLLOWER
I20260926 20:14:47.987500 12433 consensus_queue.cc:260] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77ac040e6d54b6ab4e41315bd9d0537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44907 } }
I20260926 20:14:47.987650 12433 raft_consensus.cc:388] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:47.987723 12433 raft_consensus.cc:482] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:47.987854 12433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:47.988722 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:47.980926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:56805 (local address 127.0.0.1:44907)
0926 20:14:47.981380 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:47.981395 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:47.981429 (+ 34us) server_negotiation.cc:407] Connection header received
0926 20:14:47.982178 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:47.982197 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:47.982543 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:47.982875 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:47.983283 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:47.984271 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:47.985179 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:47.985652 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:47.985814 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:47.986738 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:47.986743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:47.986756 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:47.987236 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:47.987322 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:47.987358 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:47.987505 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:47.987681 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 20:14:47.987987 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260926 20:14:47.988751 12433 raft_consensus.cc:504] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77ac040e6d54b6ab4e41315bd9d0537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44907 } }
I20260926 20:14:47.989111 12433 leader_election.cc:302] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a77ac040e6d54b6ab4e41315bd9d0537; no voters:
I20260926 20:14:47.989439 12433 leader_election.cc:288] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:47.989508 12438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:47.989699 12438 raft_consensus.cc:686] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [term 1 LEADER]: Becoming Leader. State: Replica: a77ac040e6d54b6ab4e41315bd9d0537, State: Running, Role: LEADER
I20260926 20:14:47.990074 12438 consensus_queue.cc:237] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77ac040e6d54b6ab4e41315bd9d0537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44907 } }
I20260926 20:14:47.990759 12433 sys_catalog.cc:564] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:47.991783 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:47.991954 12440 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a77ac040e6d54b6ab4e41315bd9d0537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77ac040e6d54b6ab4e41315bd9d0537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44907 } } }
I20260926 20:14:47.991981 12439 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a77ac040e6d54b6ab4e41315bd9d0537. Latest consensus state: current_term: 1 leader_uuid: "a77ac040e6d54b6ab4e41315bd9d0537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77ac040e6d54b6ab4e41315bd9d0537" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44907 } } }
I20260926 20:14:47.992063 12440 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:47.992110 12439 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:47.992492 12447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:47.995954 12447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:48.000471 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:47.994759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:54637 (local address 127.0.0.1:44907)
0926 20:14:47.994880 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:47.994884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:47.995707 (+ 823us) server_negotiation.cc:407] Connection header received
0926 20:14:47.995833 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:47.995836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:47.995888 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:47.995984 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:47.996374 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:47.997157 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:47.998426 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:47.998790 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:47.998872 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:47.999958 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:47.999962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:47.999965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:47.999979 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:48.000008 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:48.000022 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:48.000123 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:48.000282 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 20:14:48.000317 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:14:48.001827 12447 catalog_manager.cc:1347] Generated new cluster ID: fdf0316a2c8c4351861af2901834ba1b
I20260926 20:14:48.001929 12447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:48.026355 12447 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:48.027700 12447 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:48.035215 12447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a77ac040e6d54b6ab4e41315bd9d0537: Generated new TSK 0
I20260926 20:14:48.035928 12447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:48.048614 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 12365
W20260926 20:14:48.054539 12350 connection.cc:535] client connection to 127.0.0.1:44907 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44907): Connection reset by peer (error 104)
2026-09-26T20:14:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:48 dist-test-slave-7034 krb5kdc[12464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:48 dist-test-slave-7034 krb5kdc[12464](info): set up 2 sockets
Sep 26 20:14:48 dist-test-slave-7034 krb5kdc[12464](info): commencing operation
krb5kdc: starting...
W20260926 20:14:49.082567 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:49 dist-test-slave-7034 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:14:49.205557 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:49.314896 12477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:49.315187 12477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:49.315263 12477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:49.319284 12477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:49.319360 12477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:49.319398 12477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:49.319448 12477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:49.319485 12477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:49.324697 12477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:49.325723 12477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:49.326776 12477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:49.335117 12485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:49.335218 12482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:49.335273 12483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:49.335213 12477 server_base.cc:1034] running on GCE node
I20260926 20:14:49.335976 12477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:49.337128 12477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:49.338284 12477 hybrid_clock.cc:648] HybridClock initialized: now 1790453689338252 us; error 45 us; skew 500 ppm
Sep 26 20:14:49 dist-test-slave-7034 krb5kdc[12464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:49.346714 12477 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:14:49.347723 12477 webserver.cc:460] Webserver started at http://127.0.0.1:42985/ using document root <none> and password file <none>
I20260926 20:14:49.348325 12477 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:49.348418 12477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:49.348652 12477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:49.350975 12477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "df6a590555f841248d1cb2bd129701d8"
format_stamp: "Formatted at 2026-09-26 20:14:49 on dist-test-slave-7034"
server_key: "00e6bd43ec01acea31f752d825c504ad"
server_key_iv: "e3952de9728ddefc3765db0df725057b"
server_key_version: "encryptionkey@0"
I20260926 20:14:49.351557 12477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "df6a590555f841248d1cb2bd129701d8"
format_stamp: "Formatted at 2026-09-26 20:14:49 on dist-test-slave-7034"
server_key: "00e6bd43ec01acea31f752d825c504ad"
server_key_iv: "e3952de9728ddefc3765db0df725057b"
server_key_version: "encryptionkey@0"
I20260926 20:14:49.355824 12477 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:14:49.358408 12492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:49.359611 12477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:49.359750 12477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "df6a590555f841248d1cb2bd129701d8"
format_stamp: "Formatted at 2026-09-26 20:14:49 on dist-test-slave-7034"
server_key: "00e6bd43ec01acea31f752d825c504ad"
server_key_iv: "e3952de9728ddefc3765db0df725057b"
server_key_version: "encryptionkey@0"
I20260926 20:14:49.359874 12477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:49.364619 12477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:49.367928 12477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:49.368124 12477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:49.375327 12477 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44793
I20260926 20:14:49.375355 12544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44793 every 8 connection(s)
I20260926 20:14:49.376235 12477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:49.378875 12545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:49.381441 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 12477
I20260926 20:14:49.381572 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:49.381831 23754 external_mini_cluster.cc:1428] Setting key 2acc9769c62b86c01bdd78f20fef2e87
I20260926 20:14:49.384723 12545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8: Bootstrap starting.
I20260926 20:14:49.386957 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:49.383241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:44489 (local address 127.0.0.1:44793)
0926 20:14:49.383679 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:49.383693 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:49.383728 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:14:49.384333 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:49.384349 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:49.384626 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:49.384983 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:49.385744 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:49.385748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:49.385759 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:49.385770 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:49.385828 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:49.385888 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:49.386040 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:49.386113 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 20:14:49.386287 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":120,"threads_started":1}
I20260926 20:14:49.387202 12545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:49.388003 12545 log.cc:824] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:49.389814 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:49.390196 12545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8: No bootstrap required, opened a new log
I20260926 20:14:49.393450 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:49.391795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:50873 (local address 127.0.0.1:44793)
0926 20:14:49.391918 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:49.391922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:49.392104 (+ 182us) server_negotiation.cc:407] Connection header received
0926 20:14:49.392232 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:49.392234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:49.392261 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:49.392329 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:49.392998 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:49.393001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:49.393003 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:49.393013 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:49.393037 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:49.393047 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:49.393172 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:49.393294 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 20:14:49.393325 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 20:14:49.394555 12545 raft_consensus.cc:348] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6a590555f841248d1cb2bd129701d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44793 } }
I20260926 20:14:49.394764 12545 raft_consensus.cc:374] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:49.394804 12545 raft_consensus.cc:729] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a590555f841248d1cb2bd129701d8, State: Initialized, Role: FOLLOWER
I20260926 20:14:49.395143 12545 consensus_queue.cc:260] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6a590555f841248d1cb2bd129701d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44793 } }
I20260926 20:14:49.395246 12545 raft_consensus.cc:388] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:49.395282 12545 raft_consensus.cc:482] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:49.395352 12545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:49.396155 12545 raft_consensus.cc:504] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6a590555f841248d1cb2bd129701d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44793 } }
I20260926 20:14:49.396441 12545 leader_election.cc:302] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df6a590555f841248d1cb2bd129701d8; no voters:
I20260926 20:14:49.396704 12545 leader_election.cc:288] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:49.396795 12555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:49.397033 12555 raft_consensus.cc:686] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [term 1 LEADER]: Becoming Leader. State: Replica: df6a590555f841248d1cb2bd129701d8, State: Running, Role: LEADER
I20260926 20:14:49.397385 12555 consensus_queue.cc:237] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6a590555f841248d1cb2bd129701d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44793 } }
I20260926 20:14:49.397921 12545 sys_catalog.cc:564] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:49.399132 12556 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df6a590555f841248d1cb2bd129701d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6a590555f841248d1cb2bd129701d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44793 } } }
I20260926 20:14:49.399145 12557 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df6a590555f841248d1cb2bd129701d8. Latest consensus state: current_term: 1 leader_uuid: "df6a590555f841248d1cb2bd129701d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6a590555f841248d1cb2bd129701d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44793 } } }
I20260926 20:14:49.399263 12556 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:49.399333 12557 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:49.399674 12564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:49.402750 12564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:49.408092 12564 catalog_manager.cc:1347] Generated new cluster ID: e03548fc86374a2691e1f05c6218b809
I20260926 20:14:49.408154 12564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:49.430557 12564 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:49.430922 12564 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:49.444126 12564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df6a590555f841248d1cb2bd129701d8: Generated new TSK 0
I20260926 20:14:49.444918 12564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:49.449133 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 12477
2026-09-26T20:14:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:49 dist-test-slave-7034 krb5kdc[12576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:49 dist-test-slave-7034 krb5kdc[12576](info): set up 2 sockets
Sep 26 20:14:49 dist-test-slave-7034 krb5kdc[12576](info): commencing operation
krb5kdc: starting...
W20260926 20:14:50.494644 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:50 dist-test-slave-7034 krb5kdc[12576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:14:50.616840 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:50.727281 12589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:50.727555 12589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:50.727595 12589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:50.731683 12589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:50.731742 12589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:50.731762 12589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:50.731779 12589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:50.731842 12589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:50.737123 12589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38725
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:50.738205 12589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:50.739240 12589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:50.747875 12597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:50.747874 12595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:50.747874 12594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:50.748334 12589 server_base.cc:1034] running on GCE node
I20260926 20:14:50.748945 12589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:50.750171 12589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:50.751348 12589 hybrid_clock.cc:648] HybridClock initialized: now 1790453690751331 us; error 35 us; skew 500 ppm
Sep 26 20:14:50 dist-test-slave-7034 krb5kdc[12576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:50.754290 12589 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:14:50.755281 12589 webserver.cc:460] Webserver started at http://127.0.0.1:44567/ using document root <none> and password file <none>
I20260926 20:14:50.755863 12589 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:50.755926 12589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:50.756157 12589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:50.758333 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "fcaebfe86d15427b80d1c993150be337"
format_stamp: "Formatted at 2026-09-26 20:14:50 on dist-test-slave-7034"
server_key: "0732bac869c67b630960d294bc428ce2"
server_key_iv: "2814a4c07a87a2e7dc46d3e7b5804e81"
server_key_version: "encryptionkey@0"
I20260926 20:14:50.758924 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "fcaebfe86d15427b80d1c993150be337"
format_stamp: "Formatted at 2026-09-26 20:14:50 on dist-test-slave-7034"
server_key: "0732bac869c67b630960d294bc428ce2"
server_key_iv: "2814a4c07a87a2e7dc46d3e7b5804e81"
server_key_version: "encryptionkey@0"
I20260926 20:14:50.762743 12589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 20:14:50.765117 12604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:50.766433 12589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 20:14:50.766558 12589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "fcaebfe86d15427b80d1c993150be337"
format_stamp: "Formatted at 2026-09-26 20:14:50 on dist-test-slave-7034"
server_key: "0732bac869c67b630960d294bc428ce2"
server_key_iv: "2814a4c07a87a2e7dc46d3e7b5804e81"
server_key_version: "encryptionkey@0"
I20260926 20:14:50.766657 12589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:50.780368 12589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:50.783246 12589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:50.783442 12589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:50.790356 12589 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38725
I20260926 20:14:50.790382 12656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38725 every 8 connection(s)
I20260926 20:14:50.791292 12589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:50.792670 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 12589
I20260926 20:14:50.792822 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:50.793138 23754 external_mini_cluster.cc:1428] Setting key 2d1890e243ec5149234af8be9668a6c8
I20260926 20:14:50.794396 12657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:50.799937 12657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337: Bootstrap starting.
Sep 26 20:14:50 dist-test-slave-7034 krb5kdc[12576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 20:14:50.802556 12657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:50.803453 12657 log.cc:824] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:50.805675 12657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337: No bootstrap required, opened a new log
I20260926 20:14:50.807524 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:50.794528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:40035 (local address 127.0.0.1:38725)
0926 20:14:50.794950 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:50.794966 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:50.795001 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:14:50.795622 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:50.795637 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:50.795976 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:50.796300 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:50.797119 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:50.798177 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:50.799042 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:50.799511 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:50.799679 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:50.802443 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:50.802448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:50.802462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:50.802489 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:50.804383 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:50.804941 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:50.804945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:50.804949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:50.805054 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:50.805383 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:50.805385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:50.805386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:50.805779 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:50.806008 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:50.806295 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 20:14:50.806528 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":144,"threads_started":1}
I20260926 20:14:50.809203 12657 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcaebfe86d15427b80d1c993150be337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260926 20:14:50.809484 12657 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:50.809548 12657 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcaebfe86d15427b80d1c993150be337, State: Initialized, Role: FOLLOWER
I20260926 20:14:50.810042 12657 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcaebfe86d15427b80d1c993150be337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260926 20:14:50.810215 12657 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:50.810285 12657 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:50.810391 12657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:50.810513 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:50.811352 12657 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcaebfe86d15427b80d1c993150be337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260926 20:14:50.811712 12657 leader_election.cc:302] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcaebfe86d15427b80d1c993150be337; no voters:
I20260926 20:14:50.812045 12657 leader_election.cc:288] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:50.812112 12666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:50.812326 12666 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [term 1 LEADER]: Becoming Leader. State: Replica: fcaebfe86d15427b80d1c993150be337, State: Running, Role: LEADER
I20260926 20:14:50.812669 12666 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcaebfe86d15427b80d1c993150be337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } }
I20260926 20:14:50.813334 12657 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:50.814450 12667 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcaebfe86d15427b80d1c993150be337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcaebfe86d15427b80d1c993150be337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } } }
I20260926 20:14:50.814589 12667 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:50.814924 12668 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcaebfe86d15427b80d1c993150be337. Latest consensus state: current_term: 1 leader_uuid: "fcaebfe86d15427b80d1c993150be337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcaebfe86d15427b80d1c993150be337" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38725 } } }
I20260926 20:14:50.815009 12676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:50.815078 12668 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:50.818254 12676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:50.823649 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:50.812476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:34337 (local address 127.0.0.1:38725)
0926 20:14:50.812568 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:50.812572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:50.813111 (+ 539us) server_negotiation.cc:407] Connection header received
0926 20:14:50.813259 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:50.813262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:50.813317 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:50.813394 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:50.816034 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:50.816860 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:50.817693 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:50.818082 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:50.818173 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:50.819707 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:50.819711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:50.819714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:50.819746 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:50.821494 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:50.822189 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:50.822192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:50.822194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:50.822285 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:50.822678 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:50.822681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:50.822683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:50.822911 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:50.823061 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:50.823386 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 20:14:50.823488 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 20:14:50.824774 12676 catalog_manager.cc:1347] Generated new cluster ID: 208fd0d074634672bcbcb1de4ac2e058
I20260926 20:14:50.824863 12676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:50.873438 12676 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:50.874476 12676 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:50.879082 12676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcaebfe86d15427b80d1c993150be337: Generated new TSK 0
I20260926 20:14:50.879890 12676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:50.894438 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 12589
W20260926 20:14:50.899948 12574 connection.cc:535] client connection to 127.0.0.1:38725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38725): Connection reset by peer (error 104)
2026-09-26T20:14:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:50 dist-test-slave-7034 krb5kdc[12688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:50 dist-test-slave-7034 krb5kdc[12688](info): set up 2 sockets
Sep 26 20:14:50 dist-test-slave-7034 krb5kdc[12688](info): commencing operation
krb5kdc: starting...
W20260926 20:14:51.966557 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:52 dist-test-slave-7034 krb5kdc[12688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:14:52.092700 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:52.205541 12701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:52.205878 12701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:52.205924 12701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:52.210235 12701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:52.210307 12701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:52.210327 12701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:52.210345 12701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:52.210361 12701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:52.215996 12701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:52.217094 12701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:52.218230 12701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:52.226683 12707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:52.226677 12706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:52.226677 12709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:52.226907 12701 server_base.cc:1034] running on GCE node
I20260926 20:14:52.227547 12701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:52.228737 12701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:52.229918 12701 hybrid_clock.cc:648] HybridClock initialized: now 1790453692229885 us; error 47 us; skew 500 ppm
Sep 26 20:14:52 dist-test-slave-7034 krb5kdc[12688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:52.232803 12701 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:14:52.233767 12701 webserver.cc:460] Webserver started at http://127.0.0.1:42113/ using document root <none> and password file <none>
I20260926 20:14:52.234400 12701 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:52.234496 12701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:52.234728 12701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:52.236850 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "bb1a8fec19a4462c97b5f45dd739230b"
format_stamp: "Formatted at 2026-09-26 20:14:52 on dist-test-slave-7034"
server_key: "88e9bc1e4edb24736f41cf19938dd453"
server_key_iv: "361e1471b8fa3a9c4e2aab00cf77b58b"
server_key_version: "encryptionkey@0"
I20260926 20:14:52.237442 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "bb1a8fec19a4462c97b5f45dd739230b"
format_stamp: "Formatted at 2026-09-26 20:14:52 on dist-test-slave-7034"
server_key: "88e9bc1e4edb24736f41cf19938dd453"
server_key_iv: "361e1471b8fa3a9c4e2aab00cf77b58b"
server_key_version: "encryptionkey@0"
I20260926 20:14:52.241544 12701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:14:52.244148 12716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:52.245456 12701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 20:14:52.245605 12701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "bb1a8fec19a4462c97b5f45dd739230b"
format_stamp: "Formatted at 2026-09-26 20:14:52 on dist-test-slave-7034"
server_key: "88e9bc1e4edb24736f41cf19938dd453"
server_key_iv: "361e1471b8fa3a9c4e2aab00cf77b58b"
server_key_version: "encryptionkey@0"
I20260926 20:14:52.245724 12701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:52.256707 12701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:52.260205 12701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:52.260417 12701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:52.267174 12701 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41261
I20260926 20:14:52.267223 12768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41261 every 8 connection(s)
I20260926 20:14:52.268172 12701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:52.268731 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 12701
I20260926 20:14:52.268886 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:52.269183 23754 external_mini_cluster.cc:1428] Setting key a2c3963464f10e59456be533b9a7fe79
I20260926 20:14:52.271391 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:52.277311 12769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b: Bootstrap starting.
I20260926 20:14:52.278242 12772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:52.270568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:38765 (local address 127.0.0.1:41261)
0926 20:14:52.271010 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:52.271025 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:52.271061 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:14:52.271803 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:52.271819 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:52.272159 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:52.272498 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:52.272894 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:52.273974 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:52.274863 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:52.275345 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:52.275507 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:52.276462 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:52.276467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:52.276479 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:52.276867 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:52.276919 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:52.276941 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:52.277082 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:52.277266 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 20:14:52.277431 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":152,"threads_started":1}
I20260926 20:14:52.279922 12769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:52.280824 12769 log.cc:824] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:52.281518 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:52.283020 12769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b: No bootstrap required, opened a new log
I20260926 20:14:52.286760 12769 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1a8fec19a4462c97b5f45dd739230b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41261 } }
I20260926 20:14:52.287065 12769 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:52.287122 12769 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1a8fec19a4462c97b5f45dd739230b, State: Initialized, Role: FOLLOWER
I20260926 20:14:52.287662 12769 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1a8fec19a4462c97b5f45dd739230b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41261 } }
I20260926 20:14:52.287866 12769 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:52.287945 12769 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:52.288117 12769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:52.288316 12772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:52.283599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:34895 (local address 127.0.0.1:41261)
0926 20:14:52.283734 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:52.283739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:52.283955 (+ 216us) server_negotiation.cc:407] Connection header received
0926 20:14:52.284084 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:52.284086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:52.284129 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:52.284203 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:52.284504 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:52.285418 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:52.286621 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:52.286849 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:52.286939 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:52.287787 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:52.287789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:52.287791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:52.287823 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:52.287854 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:52.287868 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:52.287970 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:52.288114 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 20:14:52.288153 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:14:52.289301 12769 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1a8fec19a4462c97b5f45dd739230b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41261 } }
I20260926 20:14:52.289609 12769 leader_election.cc:302] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb1a8fec19a4462c97b5f45dd739230b; no voters:
I20260926 20:14:52.289944 12769 leader_election.cc:288] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:52.290041 12779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:52.290324 12779 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [term 1 LEADER]: Becoming Leader. State: Replica: bb1a8fec19a4462c97b5f45dd739230b, State: Running, Role: LEADER
I20260926 20:14:52.290634 12779 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1a8fec19a4462c97b5f45dd739230b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41261 } }
I20260926 20:14:52.291152 12769 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:52.292241 12781 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb1a8fec19a4462c97b5f45dd739230b. Latest consensus state: current_term: 1 leader_uuid: "bb1a8fec19a4462c97b5f45dd739230b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1a8fec19a4462c97b5f45dd739230b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41261 } } }
I20260926 20:14:52.292367 12781 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:52.292726 12788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:52.292786 12780 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb1a8fec19a4462c97b5f45dd739230b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1a8fec19a4462c97b5f45dd739230b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41261 } } }
I20260926 20:14:52.292959 12780 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:52.295630 12788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:52.300944 12788 catalog_manager.cc:1347] Generated new cluster ID: 84821dab2a1044d49bccd6ab878f1ed9
I20260926 20:14:52.301014 12788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:52.318500 12788 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:52.319734 12788 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:52.331763 12788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb1a8fec19a4462c97b5f45dd739230b: Generated new TSK 0
I20260926 20:14:52.332607 12788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:52.343854 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 12701
W20260926 20:14:52.349427 12686 connection.cc:535] client connection to 127.0.0.1:41261 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41261): Connection reset by peer (error 104)
2026-09-26T20:14:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:52 dist-test-slave-7034 krb5kdc[12800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:52 dist-test-slave-7034 krb5kdc[12800](info): set up 2 sockets
Sep 26 20:14:52 dist-test-slave-7034 krb5kdc[12800](info): commencing operation
krb5kdc: starting...
W20260926 20:14:53.402529 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:53 dist-test-slave-7034 krb5kdc[12800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 20:14:53.527369 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:53.641062 12813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:53.641382 12813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:53.641467 12813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:53.645627 12813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:53.645733 12813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:53.645769 12813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:53.645812 12813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:53.645866 12813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:53.651427 12813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:53.652552 12813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:53.653614 12813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:53.662523 12819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:53.662616 12813 server_base.cc:1034] running on GCE node
W20260926 20:14:53.662500 12818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:53.662519 12821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:53.663386 12813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:53.664556 12813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:53.665782 12813 hybrid_clock.cc:648] HybridClock initialized: now 1790453693665767 us; error 30 us; skew 500 ppm
Sep 26 20:14:53 dist-test-slave-7034 krb5kdc[12800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:53.668911 12813 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 20:14:53.669827 12813 webserver.cc:460] Webserver started at http://127.0.0.1:43727/ using document root <none> and password file <none>
I20260926 20:14:53.670471 12813 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:53.670565 12813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:53.670811 12813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:53.672989 12813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "541a6ca604ab4211b911da1a28f31c04"
format_stamp: "Formatted at 2026-09-26 20:14:53 on dist-test-slave-7034"
server_key: "040b85b04cafe510cf2d4f06681dcba6"
server_key_iv: "66120e9a9b7cdb325f27553027a00601"
server_key_version: "encryptionkey@0"
I20260926 20:14:53.673571 12813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "541a6ca604ab4211b911da1a28f31c04"
format_stamp: "Formatted at 2026-09-26 20:14:53 on dist-test-slave-7034"
server_key: "040b85b04cafe510cf2d4f06681dcba6"
server_key_iv: "66120e9a9b7cdb325f27553027a00601"
server_key_version: "encryptionkey@0"
I20260926 20:14:53.677397 12813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260926 20:14:53.680202 12828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:53.681731 12813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 20:14:53.681931 12813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "541a6ca604ab4211b911da1a28f31c04"
format_stamp: "Formatted at 2026-09-26 20:14:53 on dist-test-slave-7034"
server_key: "040b85b04cafe510cf2d4f06681dcba6"
server_key_iv: "66120e9a9b7cdb325f27553027a00601"
server_key_version: "encryptionkey@0"
I20260926 20:14:53.682065 12813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:53.687280 12813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:53.690487 12813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:53.690716 12813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:53.698074 12813 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39815
I20260926 20:14:53.698092 12880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39815 every 8 connection(s)
I20260926 20:14:53.699074 12813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:53.701962 12881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:53.703500 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 12813
I20260926 20:14:53.703665 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:53.703963 23754 external_mini_cluster.cc:1428] Setting key 2e21af9a6685cf3ae507652c4237e18c
I20260926 20:14:53.708977 12881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04: Bootstrap starting.
I20260926 20:14:53.709834 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:53.705438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:60649 (local address 127.0.0.1:39815)
0926 20:14:53.705905 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:53.705927 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:53.705972 (+ 45us) server_negotiation.cc:407] Connection header received
0926 20:14:53.706842 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:53.706867 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:53.707244 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:53.707636 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:53.708425 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:53.708430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:53.708446 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:53.708458 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:53.708522 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:53.708548 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:53.708699 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:53.708768 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 20:14:53.708928 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":115,"threads_started":1}
I20260926 20:14:53.711621 12881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:53.712500 12881 log.cc:824] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:53.712812 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:53.714610 12881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04: No bootstrap required, opened a new log
I20260926 20:14:53.716604 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:53.715000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.150:49185 (local address 127.0.0.1:39815)
0926 20:14:53.715124 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:53.715129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:53.715304 (+ 175us) server_negotiation.cc:407] Connection header received
0926 20:14:53.715417 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:53.715421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:53.715458 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:53.715535 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:53.716243 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:53.716246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:53.716249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:53.716261 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:53.716291 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:53.716305 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:53.716400 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:53.716448 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 20:14:53.716489 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:14:53.718974 12881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541a6ca604ab4211b911da1a28f31c04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39815 } }
I20260926 20:14:53.719209 12881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:53.719250 12881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 541a6ca604ab4211b911da1a28f31c04, State: Initialized, Role: FOLLOWER
I20260926 20:14:53.719679 12881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541a6ca604ab4211b911da1a28f31c04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39815 } }
I20260926 20:14:53.719805 12881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:53.719846 12881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:53.719913 12881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:53.720811 12881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541a6ca604ab4211b911da1a28f31c04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39815 } }
I20260926 20:14:53.721107 12881 leader_election.cc:302] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 541a6ca604ab4211b911da1a28f31c04; no voters:
I20260926 20:14:53.721402 12881 leader_election.cc:288] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:53.721554 12891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:53.721789 12891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [term 1 LEADER]: Becoming Leader. State: Replica: 541a6ca604ab4211b911da1a28f31c04, State: Running, Role: LEADER
I20260926 20:14:53.722158 12891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541a6ca604ab4211b911da1a28f31c04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39815 } }
I20260926 20:14:53.722676 12881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:53.723737 12893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 541a6ca604ab4211b911da1a28f31c04. Latest consensus state: current_term: 1 leader_uuid: "541a6ca604ab4211b911da1a28f31c04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541a6ca604ab4211b911da1a28f31c04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39815 } } }
I20260926 20:14:53.723861 12893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:53.724263 12900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:53.724284 12892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "541a6ca604ab4211b911da1a28f31c04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541a6ca604ab4211b911da1a28f31c04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39815 } } }
I20260926 20:14:53.724489 12892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:53.727869 12900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:53.734234 12900 catalog_manager.cc:1347] Generated new cluster ID: 09404950cb674b4abfe9fbf17fda40fe
I20260926 20:14:53.734346 12900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:53.745188 12900 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:53.745554 12900 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:53.753444 12900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 541a6ca604ab4211b911da1a28f31c04: Generated new TSK 0
I20260926 20:14:53.754222 12900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:53.761437 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 12813
2026-09-26T20:14:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:53 dist-test-slave-7034 krb5kdc[12912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:53 dist-test-slave-7034 krb5kdc[12912](info): set up 2 sockets
Sep 26 20:14:53 dist-test-slave-7034 krb5kdc[12912](info): commencing operation
krb5kdc: starting...
W20260926 20:14:54.802677 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:54 dist-test-slave-7034 krb5kdc[12912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:14:54.929725 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:41423
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:41423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:55.040087 12925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:55.040405 12925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:55.040493 12925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:55.044595 12925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:55.044684 12925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:55.044723 12925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:55.044781 12925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:55.044827 12925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:55.050170 12925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:41423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:41423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:55.051237 12925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:55.052240 12925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:55.062654 12930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:55.062659 12933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:55.062842 12925 server_base.cc:1034] running on GCE node
W20260926 20:14:55.062796 12931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:55.063568 12925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:55.064700 12925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:55.065919 12925 hybrid_clock.cc:648] HybridClock initialized: now 1790453695065875 us; error 45 us; skew 500 ppm
Sep 26 20:14:55 dist-test-slave-7034 krb5kdc[12912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453695, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:55.072150 12925 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:14:55.073086 12925 webserver.cc:460] Webserver started at http://127.23.50.190:36669/ using document root <none> and password file <none>
I20260926 20:14:55.073660 12925 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:55.073722 12925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:55.074013 12925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:55.076157 12925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "bd001916761e45f0ad3623cf0fb6ad3d"
format_stamp: "Formatted at 2026-09-26 20:14:55 on dist-test-slave-7034"
server_key: "11d4a4dc8ca0cb1f96a2115181f92bcf"
server_key_iv: "be61e35190e1663827c9b1d55330f3ef"
server_key_version: "encryptionkey@0"
I20260926 20:14:55.076730 12925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "bd001916761e45f0ad3623cf0fb6ad3d"
format_stamp: "Formatted at 2026-09-26 20:14:55 on dist-test-slave-7034"
server_key: "11d4a4dc8ca0cb1f96a2115181f92bcf"
server_key_iv: "be61e35190e1663827c9b1d55330f3ef"
server_key_version: "encryptionkey@0"
I20260926 20:14:55.080673 12925 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:14:55.083470 12940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:55.084841 12925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:14:55.084995 12925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "bd001916761e45f0ad3623cf0fb6ad3d"
format_stamp: "Formatted at 2026-09-26 20:14:55 on dist-test-slave-7034"
server_key: "11d4a4dc8ca0cb1f96a2115181f92bcf"
server_key_iv: "be61e35190e1663827c9b1d55330f3ef"
server_key_version: "encryptionkey@0"
I20260926 20:14:55.085124 12925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:55.108139 12925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:55.111475 12925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:55.111706 12925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:55.119601 12925 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:41423
I20260926 20:14:55.119635 12992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:41423 every 8 connection(s)
I20260926 20:14:55.120688 12925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:55.123633 12993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:55.126626 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 12925
I20260926 20:14:55.126797 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:55.127087 23754 external_mini_cluster.cc:1428] Setting key 3bfe8ef6a68ae135bc883b7babd301e5
I20260926 20:14:55.130060 12993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d: Bootstrap starting.
I20260926 20:14:55.132835 12993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:55.133666 12993 log.cc:824] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:55.135891 12993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d: No bootstrap required, opened a new log
Sep 26 20:14:55 dist-test-slave-7034 krb5kdc[12912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:55.139878 12993 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd001916761e45f0ad3623cf0fb6ad3d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41423 } }
I20260926 20:14:55.140168 12993 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:55.140223 12993 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd001916761e45f0ad3623cf0fb6ad3d, State: Initialized, Role: FOLLOWER
I20260926 20:14:55.140723 12993 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd001916761e45f0ad3623cf0fb6ad3d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41423 } }
I20260926 20:14:55.140887 12993 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:55.141026 12993 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:55.141151 12993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:55.142155 12993 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd001916761e45f0ad3623cf0fb6ad3d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41423 } }
I20260926 20:14:55.142506 12993 leader_election.cc:302] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd001916761e45f0ad3623cf0fb6ad3d; no voters:
I20260926 20:14:55.142902 12993 leader_election.cc:288] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:55.143004 12998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:55.143209 12998 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [term 1 LEADER]: Becoming Leader. State: Replica: bd001916761e45f0ad3623cf0fb6ad3d, State: Running, Role: LEADER
I20260926 20:14:55.143591 12998 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd001916761e45f0ad3623cf0fb6ad3d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41423 } }
I20260926 20:14:55.143883 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:55.128692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37853 (local address 127.23.50.190:41423)
0926 20:14:55.129147 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:55.129166 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:55.129209 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:14:55.130065 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:55.130088 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:55.130544 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:55.130914 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:55.131632 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:55.133127 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:55.134146 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:55.134739 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:55.134904 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:55.137584 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:55.137590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:55.137609 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:55.137644 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:55.140418 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:55.141030 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:55.141036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:55.141040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:55.141211 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:55.141567 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:55.141571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:55.141573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:55.142062 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:55.142358 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:55.142645 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 20:14:55.142869 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":115,"threads_started":1}
I20260926 20:14:55.144225 12993 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:55.144778 12998 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd001916761e45f0ad3623cf0fb6ad3d. Latest consensus state: current_term: 1 leader_uuid: "bd001916761e45f0ad3623cf0fb6ad3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd001916761e45f0ad3623cf0fb6ad3d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41423 } } }
I20260926 20:14:55.146647 12999 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd001916761e45f0ad3623cf0fb6ad3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd001916761e45f0ad3623cf0fb6ad3d" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 41423 } } }
I20260926 20:14:55.146765 12998 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:55.147302 12999 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:55.148343 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 20:14:55.149333 13012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:14:55.149415 13012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:14:55.149565 13014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:55.150518 13014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:55.156435 13014 catalog_manager.cc:1347] Generated new cluster ID: 4f07b3906a58467f972ba88e9b5f4452
I20260926 20:14:55.156495 13014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:55.159325 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:55.150610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44813 (local address 127.23.50.190:41423)
0926 20:14:55.150735 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:55.150742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:55.150757 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:14:55.150792 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:55.150795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:55.150831 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:55.150923 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:55.151585 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:55.152129 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:55.153204 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:55.153602 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:55.153718 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:55.155584 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:55.155588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:55.155593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:55.155619 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:55.157463 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:55.158018 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:55.158022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:55.158024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:55.158115 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:55.158446 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:55.158449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:55.158451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:55.158668 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:55.158865 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:55.159069 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 20:14:55.159168 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 20:14:55.165220 13014 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:55.166237 13014 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:55.173800 13014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd001916761e45f0ad3623cf0fb6ad3d: Generated new TSK 0
I20260926 20:14:55.174499 13014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:55.178535 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 12925
W20260926 20:14:55.184283 12910 connection.cc:535] client connection to 127.23.50.190:41423 recv error: Network error: failed to read from TLS socket (remote: 127.23.50.190:41423): Connection reset by peer (error 104)
2026-09-26T20:14:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:55 dist-test-slave-7034 krb5kdc[13024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:55 dist-test-slave-7034 krb5kdc[13024](info): set up 2 sockets
Sep 26 20:14:55 dist-test-slave-7034 krb5kdc[13024](info): commencing operation
krb5kdc: starting...
W20260926 20:14:56.226753 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:56 dist-test-slave-7034 krb5kdc[13024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:14:56.352983 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:45787
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:45787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:56.464154 13037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:56.464478 13037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:56.464560 13037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:56.468710 13037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:56.468813 13037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:56.468854 13037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:56.468914 13037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:56.468955 13037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:56.474411 13037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:45787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:45787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:56.475478 13037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:56.476495 13037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:56.485168 13042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:56.485168 13043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:56.485174 13045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:56.485534 13037 server_base.cc:1034] running on GCE node
I20260926 20:14:56.486212 13037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:56.487349 13037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:56.488543 13037 hybrid_clock.cc:648] HybridClock initialized: now 1790453696488530 us; error 30 us; skew 500 ppm
Sep 26 20:14:56 dist-test-slave-7034 krb5kdc[13024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453696, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:56.491254 13037 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:14:56.492087 13037 webserver.cc:460] Webserver started at http://127.23.50.190:44349/ using document root <none> and password file <none>
I20260926 20:14:56.492632 13037 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:56.492722 13037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:56.492970 13037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:56.495041 13037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "8ab949d028be4c9586ffcf3eb43d95d5"
format_stamp: "Formatted at 2026-09-26 20:14:56 on dist-test-slave-7034"
server_key: "0d682c75f64b07babe0c5046690be901"
server_key_iv: "748682dd2ec5416ad0e2b9b82311eb2d"
server_key_version: "encryptionkey@0"
I20260926 20:14:56.495601 13037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "8ab949d028be4c9586ffcf3eb43d95d5"
format_stamp: "Formatted at 2026-09-26 20:14:56 on dist-test-slave-7034"
server_key: "0d682c75f64b07babe0c5046690be901"
server_key_iv: "748682dd2ec5416ad0e2b9b82311eb2d"
server_key_version: "encryptionkey@0"
I20260926 20:14:56.499266 13037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 20:14:56.501766 13052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:56.503175 13037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:56.503325 13037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "8ab949d028be4c9586ffcf3eb43d95d5"
format_stamp: "Formatted at 2026-09-26 20:14:56 on dist-test-slave-7034"
server_key: "0d682c75f64b07babe0c5046690be901"
server_key_iv: "748682dd2ec5416ad0e2b9b82311eb2d"
server_key_version: "encryptionkey@0"
I20260926 20:14:56.503446 13037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:56.537827 13037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:56.540974 13037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:56.541199 13037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:56.548237 13037 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:45787
I20260926 20:14:56.548259 13104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:45787 every 8 connection(s)
I20260926 20:14:56.549218 13037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:56.549487 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 13037
I20260926 20:14:56.549651 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:56.550025 23754 external_mini_cluster.cc:1428] Setting key 2742065fdc612d9094267a6c4321c32b
I20260926 20:14:56.552685 13105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:56.560173 13105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5: Bootstrap starting.
I20260926 20:14:56.560359 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:56.551458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35005 (local address 127.23.50.190:45787)
0926 20:14:56.551883 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:56.551903 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:56.551946 (+ 43us) server_negotiation.cc:407] Connection header received
0926 20:14:56.552789 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:56.552814 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:56.553233 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:56.553593 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:56.554028 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:56.555550 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:56.556889 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:56.557492 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:56.557677 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:56.558660 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:56.558665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:56.558677 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:56.559063 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:56.559117 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:56.559139 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:56.559283 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:56.559462 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 20:14:56.559606 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":118,"threads_started":1}
I20260926 20:14:56.562899 13105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:56.563594 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:56.563822 13105 log.cc:824] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:56.566015 13105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5: No bootstrap required, opened a new log
I20260926 20:14:56.570184 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:56.565631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41405 (local address 127.23.50.190:45787)
0926 20:14:56.565763 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:56.565767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:56.565982 (+ 215us) server_negotiation.cc:407] Connection header received
0926 20:14:56.566204 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:56.566206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:56.566245 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:56.566313 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:56.566640 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:56.567151 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:56.568427 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:56.568635 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:56.568722 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:56.569608 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:56.569611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:56.569613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:56.569624 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:56.569647 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:56.569656 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:56.569727 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:56.569964 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 20:14:56.570022 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:14:56.570751 13105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab949d028be4c9586ffcf3eb43d95d5" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45787 } }
I20260926 20:14:56.571024 13105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:56.571094 13105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab949d028be4c9586ffcf3eb43d95d5, State: Initialized, Role: FOLLOWER
I20260926 20:14:56.571589 13105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab949d028be4c9586ffcf3eb43d95d5" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45787 } }
I20260926 20:14:56.571723 13105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:56.571792 13105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:56.571928 13105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:56.572865 13105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab949d028be4c9586ffcf3eb43d95d5" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45787 } }
I20260926 20:14:56.573220 13105 leader_election.cc:302] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ab949d028be4c9586ffcf3eb43d95d5; no voters:
I20260926 20:14:56.573529 13105 leader_election.cc:288] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:56.573669 13115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:56.573939 13115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8ab949d028be4c9586ffcf3eb43d95d5, State: Running, Role: LEADER
I20260926 20:14:56.574280 13115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab949d028be4c9586ffcf3eb43d95d5" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45787 } }
I20260926 20:14:56.574901 13105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:56.575838 13116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ab949d028be4c9586ffcf3eb43d95d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab949d028be4c9586ffcf3eb43d95d5" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45787 } } }
I20260926 20:14:56.575968 13116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:56.576452 13124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:56.576555 13117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ab949d028be4c9586ffcf3eb43d95d5. Latest consensus state: current_term: 1 leader_uuid: "8ab949d028be4c9586ffcf3eb43d95d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab949d028be4c9586ffcf3eb43d95d5" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 45787 } } }
I20260926 20:14:56.576692 13117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5 [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:56.579600 13124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:56.585345 13124 catalog_manager.cc:1347] Generated new cluster ID: 2a659977a40a4fc48a5664d25b98b052
I20260926 20:14:56.585415 13124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:56.593607 13124 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:56.594929 13124 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:56.600577 13124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ab949d028be4c9586ffcf3eb43d95d5: Generated new TSK 0
I20260926 20:14:56.601441 13124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:56.605758 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 13037
W20260926 20:14:56.610944 13022 connection.cc:535] client connection to 127.23.50.190:45787 recv error: Network error: failed to read from TLS socket (remote: 127.23.50.190:45787): Connection reset by peer (error 104)
2026-09-26T20:14:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:56 dist-test-slave-7034 krb5kdc[13136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:56 dist-test-slave-7034 krb5kdc[13136](info): set up 2 sockets
Sep 26 20:14:56 dist-test-slave-7034 krb5kdc[13136](info): commencing operation
krb5kdc: starting...
W20260926 20:14:57.650614 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:57 dist-test-slave-7034 krb5kdc[13136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:14:57.773442 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:34169
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:34169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:57.882352 13149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:57.882655 13149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:57.882728 13149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:57.886682 13149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:57.886775 13149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:57.886811 13149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:57.886863 13149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:57.886904 13149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:57.892369 13149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:34169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:34169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:57.893460 13149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:57.894520 13149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:57.902904 13155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:57.902906 13157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:57.902904 13154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:57.903096 13149 server_base.cc:1034] running on GCE node
I20260926 20:14:57.903792 13149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:57.904824 13149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:57.906008 13149 hybrid_clock.cc:648] HybridClock initialized: now 1790453697905987 us; error 31 us; skew 500 ppm
Sep 26 20:14:57 dist-test-slave-7034 krb5kdc[13136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453697, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:57.908806 13149 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:14:57.909696 13149 webserver.cc:460] Webserver started at http://127.23.50.190:34289/ using document root <none> and password file <none>
I20260926 20:14:57.910277 13149 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:57.910348 13149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:57.910516 13149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:57.912492 13149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "758caec7e6d3484d8095010af072b07b"
format_stamp: "Formatted at 2026-09-26 20:14:57 on dist-test-slave-7034"
server_key: "bfc318bc50f47fccbb3ef6b19706d213"
server_key_iv: "52a8bcce6f4c261800a377956b5d005f"
server_key_version: "encryptionkey@0"
I20260926 20:14:57.912994 13149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "758caec7e6d3484d8095010af072b07b"
format_stamp: "Formatted at 2026-09-26 20:14:57 on dist-test-slave-7034"
server_key: "bfc318bc50f47fccbb3ef6b19706d213"
server_key_iv: "52a8bcce6f4c261800a377956b5d005f"
server_key_version: "encryptionkey@0"
I20260926 20:14:57.916615 13149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 20:14:57.918983 13164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:57.920226 13149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 20:14:57.920377 13149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "758caec7e6d3484d8095010af072b07b"
format_stamp: "Formatted at 2026-09-26 20:14:57 on dist-test-slave-7034"
server_key: "bfc318bc50f47fccbb3ef6b19706d213"
server_key_iv: "52a8bcce6f4c261800a377956b5d005f"
server_key_version: "encryptionkey@0"
I20260926 20:14:57.920472 13149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:57.925148 13149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:57.928469 13149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:57.928674 13149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:57.936021 13149 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:34169
I20260926 20:14:57.936030 13216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:34169 every 8 connection(s)
I20260926 20:14:57.936986 13149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:57.939410 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 13149
I20260926 20:14:57.939565 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:57.939879 23754 external_mini_cluster.cc:1428] Setting key 95e932967ade55e69114dc9bbd2cf839
I20260926 20:14:57.940212 13217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:57.945716 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:57.941408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41813 (local address 127.23.50.190:34169)
0926 20:14:57.941904 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:57.941921 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:57.941961 (+ 40us) server_negotiation.cc:407] Connection header received
0926 20:14:57.942987 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:57.943007 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:57.943297 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:57.943586 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:57.944504 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:57.944508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:57.944521 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:57.944531 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:57.944585 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:57.944604 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:57.944715 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:57.944841 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 20:14:57.945017 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":173,"threads_started":1}
I20260926 20:14:57.947233 13217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b: Bootstrap starting.
I20260926 20:14:57.948544 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:57.949692 13217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:57.950533 13217 log.cc:824] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:57.952292 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:57.950566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57207 (local address 127.23.50.190:34169)
0926 20:14:57.950698 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:57.950703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:57.950961 (+ 258us) server_negotiation.cc:407] Connection header received
0926 20:14:57.951086 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:57.951089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:57.951127 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:57.951219 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:57.951899 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:57.951902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:57.951904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:14:57.951914 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:57.951938 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:14:57.951952 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:57.952018 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:57.952123 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 20:14:57.952156 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 20:14:57.952908 13217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b: No bootstrap required, opened a new log
I20260926 20:14:57.956393 13217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758caec7e6d3484d8095010af072b07b" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 34169 } }
I20260926 20:14:57.956640 13217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:57.956701 13217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758caec7e6d3484d8095010af072b07b, State: Initialized, Role: FOLLOWER
I20260926 20:14:57.957152 13217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758caec7e6d3484d8095010af072b07b" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 34169 } }
I20260926 20:14:57.957329 13217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:57.957409 13217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:57.957507 13217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:57.958477 13217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758caec7e6d3484d8095010af072b07b" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 34169 } }
I20260926 20:14:57.958822 13217 leader_election.cc:302] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 758caec7e6d3484d8095010af072b07b; no voters:
I20260926 20:14:57.959121 13217 leader_election.cc:288] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:57.959233 13227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:57.959476 13227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [term 1 LEADER]: Becoming Leader. State: Replica: 758caec7e6d3484d8095010af072b07b, State: Running, Role: LEADER
I20260926 20:14:57.959852 13227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758caec7e6d3484d8095010af072b07b" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 34169 } }
I20260926 20:14:57.960383 13217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:57.961509 13228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "758caec7e6d3484d8095010af072b07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758caec7e6d3484d8095010af072b07b" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 34169 } } }
I20260926 20:14:57.961649 13228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:57.962041 13236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:57.961982 13229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 758caec7e6d3484d8095010af072b07b. Latest consensus state: current_term: 1 leader_uuid: "758caec7e6d3484d8095010af072b07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758caec7e6d3484d8095010af072b07b" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 34169 } } }
I20260926 20:14:57.962306 13229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:57.965369 13236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:57.970793 13236 catalog_manager.cc:1347] Generated new cluster ID: e482cd670855457f93fa026577f592e6
I20260926 20:14:57.970861 13236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:57.984896 13236 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:57.985229 13236 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:57.991972 13236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 758caec7e6d3484d8095010af072b07b: Generated new TSK 0
I20260926 20:14:57.992673 13236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:58.002182 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 13149
2026-09-26T20:14:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:58 dist-test-slave-7034 krb5kdc[13248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:58 dist-test-slave-7034 krb5kdc[13248](info): set up 2 sockets
Sep 26 20:14:58 dist-test-slave-7034 krb5kdc[13248](info): commencing operation
krb5kdc: starting...
W20260926 20:14:59.046819 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:14:59 dist-test-slave-7034 krb5kdc[13248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:14:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:14:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:14:59.177157 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:40291
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:45585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:40291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:14:59.293339 13261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:14:59.293608 13261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:14:59.293651 13261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:14:59.297605 13261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:14:59.297667 13261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:14:59.297685 13261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:14:59.297703 13261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:14:59.297717 13261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:14:59.302914 13261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:45585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:40291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:40291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:14:59.303886 13261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:14:59.304817 13261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:14:59.313369 13269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:59.313336 13267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:14:59.313382 13266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:14:59.313426 13261 server_base.cc:1034] running on GCE node
I20260926 20:14:59.314260 13261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:14:59.315353 13261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:14:59.316548 13261 hybrid_clock.cc:648] HybridClock initialized: now 1790453699316529 us; error 34 us; skew 500 ppm
Sep 26 20:14:59 dist-test-slave-7034 krb5kdc[13248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453699, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:14:59.319437 13261 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:14:59.320271 13261 webserver.cc:460] Webserver started at http://127.23.50.190:42683/ using document root <none> and password file <none>
I20260926 20:14:59.320779 13261 fs_manager.cc:359] Metadata directory not provided
I20260926 20:14:59.320847 13261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:14:59.321007 13261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:14:59.323104 13261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "759ed213f13c436d97035c088fbdfcef"
format_stamp: "Formatted at 2026-09-26 20:14:59 on dist-test-slave-7034"
server_key: "4141053adbffe97d52e32abaa73a9eff"
server_key_iv: "66a481365cc6f0902d536551e176ff1b"
server_key_version: "encryptionkey@0"
I20260926 20:14:59.323622 13261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "759ed213f13c436d97035c088fbdfcef"
format_stamp: "Formatted at 2026-09-26 20:14:59 on dist-test-slave-7034"
server_key: "4141053adbffe97d52e32abaa73a9eff"
server_key_iv: "66a481365cc6f0902d536551e176ff1b"
server_key_version: "encryptionkey@0"
I20260926 20:14:59.327418 13261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:14:59.330030 13276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:14:59.331254 13261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:14:59.331413 13261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "759ed213f13c436d97035c088fbdfcef"
format_stamp: "Formatted at 2026-09-26 20:14:59 on dist-test-slave-7034"
server_key: "4141053adbffe97d52e32abaa73a9eff"
server_key_iv: "66a481365cc6f0902d536551e176ff1b"
server_key_version: "encryptionkey@0"
I20260926 20:14:59.331535 13261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:14:59.350847 13261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:14:59.353708 13261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:14:59.353941 13261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:14:59.361215 13261 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:40291
I20260926 20:14:59.361243 13328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:40291 every 8 connection(s)
I20260926 20:14:59.362203 13261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:14:59.363849 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 13261
I20260926 20:14:59.364003 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:14:59.364281 23754 external_mini_cluster.cc:1428] Setting key 6b6b2f10f1d5c35778c900908d10b4d5
I20260926 20:14:59.365126 13329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:14:59.370867 13329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef: Bootstrap starting.
Sep 26 20:14:59 dist-test-slave-7034 krb5kdc[13248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:14:59.373234 13329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef: Neither blocks nor log segments found. Creating new log.
I20260926 20:14:59.374120 13329 log.cc:824] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef: Log is configured to *not* fsync() on all Append() calls
I20260926 20:14:59.376242 13329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef: No bootstrap required, opened a new log
I20260926 20:14:59.378458 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:59.365692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37089 (local address 127.23.50.190:40291)
0926 20:14:59.366161 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:59.366176 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:59.366212 (+ 36us) server_negotiation.cc:407] Connection header received
0926 20:14:59.366890 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:59.366907 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:59.367248 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:59.367587 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:59.368409 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:59.369408 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:59.370351 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:59.370832 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:59.371009 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:59.373762 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:59.373767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:59.373780 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:59.373814 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:59.375635 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:59.376125 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:59.376129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:59.376134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:59.376240 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:59.376528 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:59.376530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:59.376531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:59.376921 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:59.377062 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:59.377372 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 20:14:59.377674 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":171,"threads_started":1}
I20260926 20:14:59.380018 13329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759ed213f13c436d97035c088fbdfcef" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 40291 } }
I20260926 20:14:59.380267 13329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:14:59.380327 13329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759ed213f13c436d97035c088fbdfcef, State: Initialized, Role: FOLLOWER
I20260926 20:14:59.380801 13329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759ed213f13c436d97035c088fbdfcef" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 40291 } }
I20260926 20:14:59.380965 13329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:14:59.381052 13329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:14:59.381156 13329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:14:59.381265 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:14:59.382373 13329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759ed213f13c436d97035c088fbdfcef" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 40291 } }
I20260926 20:14:59.382794 13329 leader_election.cc:302] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 759ed213f13c436d97035c088fbdfcef; no voters:
I20260926 20:14:59.383121 13329 leader_election.cc:288] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:14:59.383217 13338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:14:59.383474 13338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [term 1 LEADER]: Becoming Leader. State: Replica: 759ed213f13c436d97035c088fbdfcef, State: Running, Role: LEADER
I20260926 20:14:59.383854 13338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759ed213f13c436d97035c088fbdfcef" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 40291 } }
I20260926 20:14:59.384361 13329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:14:59.385455 13340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 759ed213f13c436d97035c088fbdfcef. Latest consensus state: current_term: 1 leader_uuid: "759ed213f13c436d97035c088fbdfcef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759ed213f13c436d97035c088fbdfcef" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 40291 } } }
I20260926 20:14:59.385836 13340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:59.385735 13339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "759ed213f13c436d97035c088fbdfcef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759ed213f13c436d97035c088fbdfcef" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 40291 } } }
I20260926 20:14:59.386140 13339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef [sys.catalog]: This master's current role is: LEADER
I20260926 20:14:59.386759 13351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:14:59.389696 13351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:14:59.393697 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:14:59.383510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38435 (local address 127.23.50.190:40291)
0926 20:14:59.383618 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 20:14:59.383623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:14:59.385151 (+ 1528us) server_negotiation.cc:407] Connection header received
0926 20:14:59.385347 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:14:59.385350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:14:59.385397 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:14:59.385453 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:14:59.386409 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:59.387134 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:59.388187 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:14:59.388522 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:14:59.388631 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:14:59.390274 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:14:59.390278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:14:59.390280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:14:59.390306 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:14:59.391960 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:59.392471 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:59.392480 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:59.392481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:59.392554 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:14:59.392821 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:14:59.392826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:14:59.392828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:14:59.393036 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:14:59.393164 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 20:14:59.393422 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 20:14:59.393539 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:14:59.395999 13351 catalog_manager.cc:1347] Generated new cluster ID: 6589565254e847158b2e5bcd6ab4ee73
I20260926 20:14:59.396065 13351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:14:59.406569 13351 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:14:59.407501 13351 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:14:59.411515 13351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 759ed213f13c436d97035c088fbdfcef: Generated new TSK 0
I20260926 20:14:59.412156 13351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:14:59.417198 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 13261
2026-09-26T20:14:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:14:59 dist-test-slave-7034 krb5kdc[13360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:14:59 dist-test-slave-7034 krb5kdc[13360](info): set up 2 sockets
Sep 26 20:14:59 dist-test-slave-7034 krb5kdc[13360](info): commencing operation
krb5kdc: starting...
W20260926 20:15:00.474716 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:15:00 dist-test-slave-7034 krb5kdc[13360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:15:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:15:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:15:00.601686 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:38619
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:38619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:00.714260 13373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:00.714553 13373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:00.714604 13373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:00.718784 13373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:00.718852 13373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:00.718873 13373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:00.718889 13373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:00.718905 13373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:00.724397 13373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:38619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:38619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:00.725517 13373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:00.726600 13373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:00.735632 13381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:00.735610 13379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:00.735620 13373 server_base.cc:1034] running on GCE node
W20260926 20:15:00.735595 13378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:00.736456 13373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:00.737588 13373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:00.738798 13373 hybrid_clock.cc:648] HybridClock initialized: now 1790453700738769 us; error 46 us; skew 500 ppm
Sep 26 20:15:00 dist-test-slave-7034 krb5kdc[13360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453700, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:00.755229 13373 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:15:00.756297 13373 webserver.cc:460] Webserver started at http://127.23.50.190:42225/ using document root <none> and password file <none>
I20260926 20:15:00.756866 13373 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:00.756932 13373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:00.757108 13373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:00.759303 13373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "2130c1159c79498595363889c3e16cc2"
format_stamp: "Formatted at 2026-09-26 20:15:00 on dist-test-slave-7034"
server_key: "f145cbc887a9f6ed597da5ca2e40eccc"
server_key_iv: "4e8aeb59d2e2575b23239b984c782159"
server_key_version: "encryptionkey@0"
I20260926 20:15:00.759838 13373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "2130c1159c79498595363889c3e16cc2"
format_stamp: "Formatted at 2026-09-26 20:15:00 on dist-test-slave-7034"
server_key: "f145cbc887a9f6ed597da5ca2e40eccc"
server_key_iv: "4e8aeb59d2e2575b23239b984c782159"
server_key_version: "encryptionkey@0"
I20260926 20:15:00.763688 13373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 20:15:00.766201 13388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:00.767465 13373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 20:15:00.767572 13373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "2130c1159c79498595363889c3e16cc2"
format_stamp: "Formatted at 2026-09-26 20:15:00 on dist-test-slave-7034"
server_key: "f145cbc887a9f6ed597da5ca2e40eccc"
server_key_iv: "4e8aeb59d2e2575b23239b984c782159"
server_key_version: "encryptionkey@0"
I20260926 20:15:00.767664 13373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:00.783378 13373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:00.786495 13373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:00.786656 13373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:00.793613 13373 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:38619
I20260926 20:15:00.793625 13440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:38619 every 8 connection(s)
I20260926 20:15:00.794624 13373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:15:00.797477 13441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:00.798483 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 13373
I20260926 20:15:00.798625 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:15:00.798950 23754 external_mini_cluster.cc:1428] Setting key db6fe1e2ad83dcc773578fe0046ac6e6
I20260926 20:15:00.803927 13441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2: Bootstrap starting.
I20260926 20:15:00.806622 13441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:00.807493 13441 log.cc:824] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:00.808444 13444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:00.800442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60325 (local address 127.23.50.190:38619)
0926 20:15:00.800885 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:00.800902 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:00.800939 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:15:00.801610 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:00.801628 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:00.802051 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:00.802366 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:00.802719 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:00.803773 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:00.804986 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:00.805458 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:00.805582 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:00.806438 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:00.806443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:00.806456 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:15:00.806912 (+ 456us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:00.806968 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:15:00.806992 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:00.807068 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:00.807227 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 20:15:00.807465 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":143,"threads_started":1}
I20260926 20:15:00.809569 13441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2: No bootstrap required, opened a new log
I20260926 20:15:00.811367 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:15:00.814237 13441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2130c1159c79498595363889c3e16cc2" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38619 } }
I20260926 20:15:00.814538 13441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:00.814612 13441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2130c1159c79498595363889c3e16cc2, State: Initialized, Role: FOLLOWER
I20260926 20:15:00.815097 13441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2130c1159c79498595363889c3e16cc2" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38619 } }
I20260926 20:15:00.815259 13441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:15:00.815325 13441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:15:00.815415 13441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:00.816414 13441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2130c1159c79498595363889c3e16cc2" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38619 } }
I20260926 20:15:00.816797 13441 leader_election.cc:302] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2130c1159c79498595363889c3e16cc2; no voters:
I20260926 20:15:00.817162 13441 leader_election.cc:288] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:15:00.817312 13451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:00.817587 13451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [term 1 LEADER]: Becoming Leader. State: Replica: 2130c1159c79498595363889c3e16cc2, State: Running, Role: LEADER
I20260926 20:15:00.818154 13451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2130c1159c79498595363889c3e16cc2" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38619 } }
I20260926 20:15:00.818303 13444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:00.813716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40601 (local address 127.23.50.190:38619)
0926 20:15:00.813866 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:00.813871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:00.814109 (+ 238us) server_negotiation.cc:407] Connection header received
0926 20:15:00.814219 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:00.814222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:00.814273 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:00.814363 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:00.814730 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:00.815506 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:00.816290 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:00.816646 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:00.816766 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:00.817646 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:00.817649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:00.817652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:15:00.817664 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:00.817693 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:15:00.817705 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:00.817775 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:00.818115 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 20:15:00.818172 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 20:15:00.818663 13441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:15:00.820022 13453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2130c1159c79498595363889c3e16cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2130c1159c79498595363889c3e16cc2" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38619 } } }
I20260926 20:15:00.820156 13453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:00.820600 13461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:15:00.820519 13452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2130c1159c79498595363889c3e16cc2. Latest consensus state: current_term: 1 leader_uuid: "2130c1159c79498595363889c3e16cc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2130c1159c79498595363889c3e16cc2" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38619 } } }
I20260926 20:15:00.821448 13452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2 [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:00.824167 13461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:15:00.829797 13461 catalog_manager.cc:1347] Generated new cluster ID: c7c7fc1a3ce9468bba435cb6e1469d00
I20260926 20:15:00.829901 13461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:15:00.856950 13461 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:15:00.858464 13461 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:15:00.866288 13461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2130c1159c79498595363889c3e16cc2: Generated new TSK 0
I20260926 20:15:00.867147 13461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:15:00.873978 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 13373
2026-09-26T20:15:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:15:00 dist-test-slave-7034 krb5kdc[13472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:15:00 dist-test-slave-7034 krb5kdc[13472](info): set up 2 sockets
Sep 26 20:15:00 dist-test-slave-7034 krb5kdc[13472](info): commencing operation
krb5kdc: starting...
W20260926 20:15:01.934751 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:15:02 dist-test-slave-7034 krb5kdc[13472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:15:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:15:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:15:02.059007 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37351
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:34997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:37351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:02.174918 13485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:02.175209 13485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:02.175252 13485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:02.179587 13485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:02.179666 13485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:02.179687 13485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:02.179703 13485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:02.179719 13485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:02.185909 13485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:34997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:37351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:37351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:02.187027 13485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:02.188009 13485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:02.197029 13491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:02.197029 13493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:02.197068 13490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:02.197506 13485 server_base.cc:1034] running on GCE node
I20260926 20:15:02.198201 13485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:02.199422 13485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:02.200618 13485 hybrid_clock.cc:648] HybridClock initialized: now 1790453702200568 us; error 65 us; skew 500 ppm
Sep 26 20:15:02 dist-test-slave-7034 krb5kdc[13472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453702, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:02.203552 13485 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:15:02.204496 13485 webserver.cc:460] Webserver started at http://127.23.50.190:45863/ using document root <none> and password file <none>
I20260926 20:15:02.205083 13485 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:02.205176 13485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:02.205416 13485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:02.207686 13485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "0bf3bd348e404a6d85a5a4911a29d5b4"
format_stamp: "Formatted at 2026-09-26 20:15:02 on dist-test-slave-7034"
server_key: "f146d3be0af3f11b9f39714bf9ebd344"
server_key_iv: "1f91c22ffa7dc4f625e6d69adb62722e"
server_key_version: "encryptionkey@0"
I20260926 20:15:02.208395 13485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "0bf3bd348e404a6d85a5a4911a29d5b4"
format_stamp: "Formatted at 2026-09-26 20:15:02 on dist-test-slave-7034"
server_key: "f146d3be0af3f11b9f39714bf9ebd344"
server_key_iv: "1f91c22ffa7dc4f625e6d69adb62722e"
server_key_version: "encryptionkey@0"
I20260926 20:15:02.212481 13485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 20:15:02.215749 13500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:02.217207 13485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 20:15:02.217387 13485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "0bf3bd348e404a6d85a5a4911a29d5b4"
format_stamp: "Formatted at 2026-09-26 20:15:02 on dist-test-slave-7034"
server_key: "f146d3be0af3f11b9f39714bf9ebd344"
server_key_iv: "1f91c22ffa7dc4f625e6d69adb62722e"
server_key_version: "encryptionkey@0"
I20260926 20:15:02.217530 13485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:02.222450 13485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:02.226099 13485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:02.226375 13485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:02.233950 13485 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:37351
I20260926 20:15:02.233979 13552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:37351 every 8 connection(s)
I20260926 20:15:02.235003 13485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:15:02.235448 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 13485
I20260926 20:15:02.235605 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:15:02.235913 23754 external_mini_cluster.cc:1428] Setting key db6cf99420d9db31b5135b61d3c1f96e
I20260926 20:15:02.238492 13553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:02.241997 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:02.237444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56099 (local address 127.23.50.190:37351)
0926 20:15:02.238015 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:02.238041 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:02.238085 (+ 44us) server_negotiation.cc:407] Connection header received
0926 20:15:02.238863 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:02.238883 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:02.239189 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:02.239550 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:02.240399 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:02.240403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:02.240415 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:15:02.240426 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:02.240480 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:15:02.240500 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:02.240666 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:02.240736 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 20:15:02.240900 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":156,"threads_started":1}
I20260926 20:15:02.245146 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:15:02.245386 13553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4: Bootstrap starting.
I20260926 20:15:02.247881 13553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:02.248842 13553 log.cc:824] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:02.249048 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:02.247294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52243 (local address 127.23.50.190:37351)
0926 20:15:02.247436 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:02.247441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:02.247646 (+ 205us) server_negotiation.cc:407] Connection header received
0926 20:15:02.247767 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:02.247769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:02.247801 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:02.247866 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:02.248629 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:02.248631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:02.248634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 20:15:02.248644 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:02.248669 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 20:15:02.248679 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:02.248743 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:02.248857 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 20:15:02.248889 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 20:15:02.250883 13553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4: No bootstrap required, opened a new log
I20260926 20:15:02.254434 13553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf3bd348e404a6d85a5a4911a29d5b4" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37351 } }
I20260926 20:15:02.254688 13553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 20:15:02.254753 13553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf3bd348e404a6d85a5a4911a29d5b4, State: Initialized, Role: FOLLOWER
I20260926 20:15:02.255218 13553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf3bd348e404a6d85a5a4911a29d5b4" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37351 } }
I20260926 20:15:02.255373 13553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 20:15:02.255442 13553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 20:15:02.255544 13553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:02.256443 13553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf3bd348e404a6d85a5a4911a29d5b4" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37351 } }
I20260926 20:15:02.256796 13553 leader_election.cc:302] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bf3bd348e404a6d85a5a4911a29d5b4; no voters:
I20260926 20:15:02.257122 13553 leader_election.cc:288] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 20:15:02.257274 13563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:02.257494 13563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [term 1 LEADER]: Becoming Leader. State: Replica: 0bf3bd348e404a6d85a5a4911a29d5b4, State: Running, Role: LEADER
I20260926 20:15:02.257894 13563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf3bd348e404a6d85a5a4911a29d5b4" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37351 } }
I20260926 20:15:02.258463 13553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:15:02.259613 13564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bf3bd348e404a6d85a5a4911a29d5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf3bd348e404a6d85a5a4911a29d5b4" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37351 } } }
I20260926 20:15:02.259654 13565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bf3bd348e404a6d85a5a4911a29d5b4. Latest consensus state: current_term: 1 leader_uuid: "0bf3bd348e404a6d85a5a4911a29d5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf3bd348e404a6d85a5a4911a29d5b4" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 37351 } } }
I20260926 20:15:02.259749 13565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:02.259749 13564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4 [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:02.260191 13572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:15:02.263270 13572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:15:02.268750 13572 catalog_manager.cc:1347] Generated new cluster ID: b0249aa20bd64d8eb6ec56a51ec6aee2
I20260926 20:15:02.268819 13572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:15:02.283694 13572 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:15:02.284034 13572 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:15:02.289346 13572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bf3bd348e404a6d85a5a4911a29d5b4: Generated new TSK 0
I20260926 20:15:02.290066 13572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 20:15:02.299232 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 13485
2026-09-26T20:15:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1425 ms)
[----------] 18 tests from AuthTokenIssuingTest (25552 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:15:02 dist-test-slave-7034 krb5kdc[13584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:15:02 dist-test-slave-7034 krb5kdc[13584](info): set up 2 sockets
Sep 26 20:15:02 dist-test-slave-7034 krb5kdc[13584](info): commencing operation
krb5kdc: starting...
W20260926 20:15:03.358733 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:15:03 dist-test-slave-7034 krb5kdc[13584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:15:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:15:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:15:03.486642 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:38339
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:38339,127.23.50.189:33707,127.23.50.188:33225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:03.602322 13597 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:15:03.602653 13597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:03.602700 13597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:03.602775 13597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:03.607184 13597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:03.607259 13597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:03.607278 13597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:03.607296 13597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:03.607312 13597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:03.613067 13597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:38339,127.23.50.189:33707,127.23.50.188:33225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:03.614313 13597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:03.615372 13597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:03.624583 13603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:03.624636 13605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:03.624639 13602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:03.624869 13597 server_base.cc:1034] running on GCE node
I20260926 20:15:03.625489 13597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:03.626811 13597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:03.628043 13597 hybrid_clock.cc:648] HybridClock initialized: now 1790453703628018 us; error 43 us; skew 500 ppm
Sep 26 20:15:03 dist-test-slave-7034 krb5kdc[13584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453703, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:03.631371 13597 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:15:03.632470 13597 webserver.cc:460] Webserver started at http://127.23.50.190:39827/ using document root <none> and password file <none>
I20260926 20:15:03.633159 13597 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:03.633267 13597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:03.633538 13597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:03.636013 13597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "a06d376c90c44b8b8fed9966f6fd8823"
format_stamp: "Formatted at 2026-09-26 20:15:03 on dist-test-slave-7034"
server_key: "cb32ffa5c02ffe6a8cae37002c378acc"
server_key_iv: "dbe950cc9076c122f9d851a7ded3008a"
server_key_version: "encryptionkey@0"
I20260926 20:15:03.636691 13597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "a06d376c90c44b8b8fed9966f6fd8823"
format_stamp: "Formatted at 2026-09-26 20:15:03 on dist-test-slave-7034"
server_key: "cb32ffa5c02ffe6a8cae37002c378acc"
server_key_iv: "dbe950cc9076c122f9d851a7ded3008a"
server_key_version: "encryptionkey@0"
I20260926 20:15:03.641010 13597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:15:03.643890 13612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:03.645430 13597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:03.645713 13597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "a06d376c90c44b8b8fed9966f6fd8823"
format_stamp: "Formatted at 2026-09-26 20:15:03 on dist-test-slave-7034"
server_key: "cb32ffa5c02ffe6a8cae37002c378acc"
server_key_iv: "dbe950cc9076c122f9d851a7ded3008a"
server_key_version: "encryptionkey@0"
I20260926 20:15:03.645849 13597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:03.670861 13597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:03.674692 13597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:03.674940 13597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:03.682720 13597 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:38339
I20260926 20:15:03.683032 13664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:38339 every 8 connection(s)
I20260926 20:15:03.683916 13597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:15:03.686908 13665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:03.690770 13665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:15:03.694083 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 13597
I20260926 20:15:03.694240 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:15:03.694547 23754 external_mini_cluster.cc:1428] Setting key e118d58fea05d440a6841d2a061da0e6
Sep 26 20:15:03 dist-test-slave-7034 krb5kdc[13584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453703, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.50.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.189@KRBTEST.COM" created.
I20260926 20:15:03.709764 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.693221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:38339 (local address 127.0.0.1:48472)
0926 20:15:03.694084 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.694125 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:03.695122 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:03.695851 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:03.695857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:03.696327 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:03.696572 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:03.696585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.698312 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.698315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:03.699322 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:03.699334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.700371 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.700375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:03.700494 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:03.701172 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:03.701191 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:03.703092 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:03.705800 (+ 2708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:03.705810 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:03.705822 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:03.706170 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:03.706597 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:03.706604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:03.706609 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:03.706791 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:03.707726 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:03.707733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:03.707928 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 20:15:03.708131 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 20:15:03.708578 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"spinlock_wait_cycles":47488,"thread_start_us":174,"threads_started":1}
I20260926 20:15:03.710382 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.693219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.23.50.190:38339)
0926 20:15:03.693779 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:03.693793 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:03.694360 (+ 567us) server_negotiation.cc:407] Connection header received
0926 20:15:03.695462 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:03.695467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:03.695649 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:03.695854 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:03.696735 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.698143 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.699472 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.700179 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.700375 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:03.703287 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:03.703296 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:03.703311 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:03.703350 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:03.705681 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:03.706314 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:03.706323 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:03.706325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:03.706475 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:03.706909 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:03.706912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:03.706914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:03.707594 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:03.707765 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:03.710028 (+ 2263us) server_negotiation.cc:299] Negotiation successful
0926 20:15:03.710182 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":174,"threads_started":1}
I20260926 20:15:03.713433 13665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:15:03.714342 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.714015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:33707 (local address 127.0.0.1:51610)
0926 20:15:03.714207 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.714281 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:33707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260926 20:15:03.714556 13616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.50.189:33707: connect: Connection refused (error 111)
I20260926 20:15:03.715605 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.715333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:33707 (local address 127.0.0.1:51612)
0926 20:15:03.715469 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.715530 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:33707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 20:15:03.715842 13665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.189:33707: Network error: Client connection negotiation failed: client connection to 127.23.50.189:33707: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.189.keytab.
Entry for principal kudu/127.23.50.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.189.keytab.
I20260926 20:15:03.726150 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.189:33707
--webserver_interface=127.23.50.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.189.keytab
--principal=kudu/127.23.50.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:38339,127.23.50.189:33707,127.23.50.188:33225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:15:03.765106 13665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } attempt: 1
I20260926 20:15:03.766094 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.765708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:33707 (local address 127.0.0.1:51626)
0926 20:15:03.765920 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.766000 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:33707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 20:15:03.766894 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.766649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:33707 (local address 127.0.0.1:51634)
0926 20:15:03.766773 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.766831 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:33707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 20:15:03.767119 13665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.189:33707: Network error: Client connection negotiation failed: client connection to 127.23.50.189:33707: connect: Connection refused (error 111)
I20260926 20:15:03.835321 13665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } attempt: 2
I20260926 20:15:03.836239 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.835905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:33707 (local address 127.0.0.1:51638)
0926 20:15:03.836071 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.836147 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:33707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 20:15:03.837028 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.836808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:33707 (local address 127.0.0.1:51650)
0926 20:15:03.836915 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.836966 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:33707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:15:03.837180 13665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.189:33707: Network error: Client connection negotiation failed: client connection to 127.23.50.189:33707: connect: Connection refused (error 111)
W20260926 20:15:03.840806 13672 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:15:03.841089 13672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:03.841143 13672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:03.841224 13672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:03.845379 13672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:03.845438 13672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:03.845458 13672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:03.845515 13672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:03.845552 13672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:03.851152 13672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:38339,127.23.50.189:33707,127.23.50.188:33225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.189.keytab
--principal=kudu/127.23.50.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.189:33707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.50.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:03.852240 13672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:03.853302 13672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:03.862112 13678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:03.862118 13680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:03.862152 13677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:03.862178 13672 server_base.cc:1034] running on GCE node
I20260926 20:15:03.863047 13672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:03.864264 13672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:03.865448 13672 hybrid_clock.cc:648] HybridClock initialized: now 1790453703865434 us; error 27 us; skew 500 ppm
Sep 26 20:15:03 dist-test-slave-7034 krb5kdc[13584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453703, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:03.868709 13672 init.cc:377] Logged in from keytab as kudu/127.23.50.189@KRBTEST.COM (short username kudu)
I20260926 20:15:03.869668 13672 webserver.cc:460] Webserver started at http://127.23.50.189:35565/ using document root <none> and password file <none>
I20260926 20:15:03.870328 13672 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:03.870397 13672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:03.870628 13672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:03.872859 13672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/data/instance:
uuid: "b82eb7ab3c72402f9abd4a983fe02c82"
format_stamp: "Formatted at 2026-09-26 20:15:03 on dist-test-slave-7034"
server_key: "0b4beed248059e7dca2eb390e7b52024"
server_key_iv: "8777ce83124feddcb4249dd4fad9ba1f"
server_key_version: "encryptionkey@0"
I20260926 20:15:03.873476 13672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/wal/instance:
uuid: "b82eb7ab3c72402f9abd4a983fe02c82"
format_stamp: "Formatted at 2026-09-26 20:15:03 on dist-test-slave-7034"
server_key: "0b4beed248059e7dca2eb390e7b52024"
server_key_iv: "8777ce83124feddcb4249dd4fad9ba1f"
server_key_version: "encryptionkey@0"
I20260926 20:15:03.877665 13672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 20:15:03.880414 13687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:03.881805 13672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:03.882019 13672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/wal
uuid: "b82eb7ab3c72402f9abd4a983fe02c82"
format_stamp: "Formatted at 2026-09-26 20:15:03 on dist-test-slave-7034"
server_key: "0b4beed248059e7dca2eb390e7b52024"
server_key_iv: "8777ce83124feddcb4249dd4fad9ba1f"
server_key_version: "encryptionkey@0"
I20260926 20:15:03.882140 13672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:03.908175 13672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:03.912140 13672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:03.912330 13672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:03.920784 13672 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.189:33707
I20260926 20:15:03.921083 13739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.189:33707 every 8 connection(s)
I20260926 20:15:03.921983 13672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
I20260926 20:15:03.923560 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 13672
I20260926 20:15:03.923720 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-1/wal/instance
I20260926 20:15:03.924023 23754 external_mini_cluster.cc:1428] Setting key 2161c4f8622fb457e00499bacd9f0a0e
I20260926 20:15:03.925567 13740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:03.928397 13665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } attempt: 3
I20260926 20:15:03.930406 13740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.50.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.188@KRBTEST.COM" created.
Sep 26 20:15:03 dist-test-slave-7034 krb5kdc[13584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453703, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.189@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
Sep 26 20:15:03 dist-test-slave-7034 krb5kdc[13584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453703, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.189@KRBTEST.COM
I20260926 20:15:03.945204 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.929006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:33707 (local address 127.0.0.1:51664)
0926 20:15:03.929284 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.929304 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:03.929428 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:03.932442 (+ 3014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:03.932448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:03.932651 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:03.932801 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:03.932813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.934852 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.934856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:03.935750 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:03.935763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.936891 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.936895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:03.937006 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:03.937747 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:03.937761 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:03.939713 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:03.943083 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:03.943086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:03.943088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:03.943316 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:03.943629 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:03.943632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:03.943636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:03.943745 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:03.944537 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:03.944543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:03.944663 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 20:15:03.944766 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 20:15:03.944970 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260926 20:15:03.945617 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.932228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48480 (local address 127.23.50.190:38339)
0926 20:15:03.932364 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:03.932370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:03.932736 (+ 366us) server_negotiation.cc:407] Connection header received
0926 20:15:03.932939 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:03.932944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:03.933001 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:03.933070 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:03.933985 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.934537 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.936042 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.936495 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.936639 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:03.940711 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:03.940714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:03.940716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:03.940739 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:03.942410 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:03.943006 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:03.943009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:03.943011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:03.943102 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:03.943655 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:03.943658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:03.943659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:03.943917 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:03.944028 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:03.945345 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0926 20:15:03.945488 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 20:15:03.946628 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.932204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:38339 (local address 127.0.0.1:48480)
0926 20:15:03.932609 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.932661 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:03.932823 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:03.933233 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:03.933237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:03.933568 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:03.933741 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:03.933752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.934679 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.934685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:03.935879 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:03.935892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.936708 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.936716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:03.936943 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:03.937691 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:03.937719 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:03.940310 (+ 2591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:03.942540 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:03.942552 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:03.942565 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:03.942894 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:03.943218 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:03.943225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:03.943229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:03.943506 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:03.944027 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:03.944036 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:03.944347 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 20:15:03.945816 (+ 1469us) client_negotiation.cc:240] Negotiation successful
0926 20:15:03.946032 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"spinlock_wait_cycles":24832,"thread_start_us":157,"threads_started":1}
I20260926 20:15:03.946635 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.929793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.23.50.189:33707)
0926 20:15:03.930366 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:03.930388 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:03.930441 (+ 53us) server_negotiation.cc:407] Connection header received
0926 20:15:03.931500 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:03.931528 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:03.932043 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:03.933061 (+ 1018us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:03.933496 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.934728 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.936035 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.936739 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.936937 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:03.939896 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:03.939907 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:03.939929 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:03.939967 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:03.942958 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:03.943427 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:03.943433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:03.943435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:03.943520 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:03.943857 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:03.943862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:03.943864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:03.944387 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:03.944551 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:03.944913 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 20:15:03.945293 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":386,"thread_start_us":158,"threads_started":1}
I20260926 20:15:03.948293 13740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:15:03.948705 13665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:15:03.949499 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.949138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59062)
0926 20:15:03.949325 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.949414 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 20:15:03.950434 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.950132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59072)
0926 20:15:03.950330 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.950378 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260926 20:15:03.950594 13665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:33225: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Sep 26 20:15:03 dist-test-slave-7034 krb5kdc[13584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453703, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.189@KRBTEST.COM for kudu/127.23.50.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.188.keytab.
Entry for principal kudu/127.23.50.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.188.keytab.
I20260926 20:15:03.957926 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.949188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:33707 (local address 127.0.0.1:51676)
0926 20:15:03.949288 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.949303 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:03.949388 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:03.949689 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:03.949691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:03.949943 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:03.950061 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:03.950068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.950924 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.950927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:03.951469 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:03.951477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.951971 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.951974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:03.952063 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:03.952588 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:03.952600 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:03.954023 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:03.956025 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:03.956028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:03.956030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:03.956264 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:03.956567 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:03.956570 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:03.956571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:03.956685 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:03.957226 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:03.957229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:03.957331 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 20:15:03.957462 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 20:15:03.957652 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 20:15:03.957926 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.949321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.23.50.189:33707)
0926 20:15:03.949444 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:03.949449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:03.949464 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:15:03.949500 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:03.949503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:03.949554 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:03.949639 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:03.950172 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.950822 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.951609 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:03.951861 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:03.951972 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:03.954208 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:03.954211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:03.954213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:03.954236 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:03.955913 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:03.956408 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:03.956411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:03.956412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:03.956492 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:03.956800 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:03.956803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:03.956805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:03.957027 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:03.957171 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:03.957519 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 20:15:03.957718 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":53}
I20260926 20:15:03.958894 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.188:33225
--webserver_interface=127.23.50.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.188.keytab
--principal=kudu/127.23.50.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:38339,127.23.50.189:33707,127.23.50.188:33225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:15:03.959332 13740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:15:03.960184 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.959769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59088)
0926 20:15:03.959950 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.960070 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 20:15:03.960449 13688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
I20260926 20:15:03.961387 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.961166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59094)
0926 20:15:03.961259 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.961314 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 20:15:03.961565 13740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:33225: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
I20260926 20:15:03.981775 13665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } attempt: 1
I20260926 20:15:03.982774 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.982461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59110)
0926 20:15:03.982618 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.982688 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 20:15:03.983522 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:03.983296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59112)
0926 20:15:03.983405 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:03.983454 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:15:03.983662 13665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:33225: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
I20260926 20:15:04.010890 13740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } attempt: 1
I20260926 20:15:04.011904 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.011543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59114)
0926 20:15:04.011723 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:04.011822 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 20:15:04.012740 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.012480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59124)
0926 20:15:04.012618 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:04.012674 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 20:15:04.012878 13740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:33225: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
I20260926 20:15:04.058823 13665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } attempt: 2
I20260926 20:15:04.059727 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.059420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59134)
0926 20:15:04.059574 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:04.059640 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 20:15:04.060537 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.060308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59148)
0926 20:15:04.060423 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:04.060479 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 20:15:04.060673 13665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:33225: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
I20260926 20:15:04.081059 13740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } attempt: 2
W20260926 20:15:04.080972 13746 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:15:04.081355 13746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:04.081418 13746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:04.081498 13746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 20:15:04.081991 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.081669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59152)
0926 20:15:04.081816 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:04.081917 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 20:15:04.082706 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.082480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59158)
0926 20:15:04.082601 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:04.082648 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 20:15:04.082839 13740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:33225: Network error: Client connection negotiation failed: client connection to 127.23.50.188:33225: connect: Connection refused (error 111)
W20260926 20:15:04.086081 13746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:04.086158 13746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:04.086205 13746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:04.086248 13746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:04.086298 13746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:04.091801 13746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:38339,127.23.50.189:33707,127.23.50.188:33225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.188.keytab
--principal=kudu/127.23.50.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.188:33225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.50.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:04.092931 13746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:04.094062 13746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:04.102531 13755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:04.102533 13752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:04.102669 13746 server_base.cc:1034] running on GCE node
W20260926 20:15:04.102548 13753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:04.103520 13746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:04.104725 13746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:04.105931 13746 hybrid_clock.cc:648] HybridClock initialized: now 1790453704105920 us; error 34 us; skew 500 ppm
Sep 26 20:15:04 dist-test-slave-7034 krb5kdc[13584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453704, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:04.108834 13746 init.cc:377] Logged in from keytab as kudu/127.23.50.188@KRBTEST.COM (short username kudu)
I20260926 20:15:04.109740 13746 webserver.cc:460] Webserver started at http://127.23.50.188:43323/ using document root <none> and password file <none>
I20260926 20:15:04.110376 13746 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:04.110481 13746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:04.110720 13746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:04.112850 13746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/data/instance:
uuid: "3bf95533f9c44c41b59e6fbb35e01e90"
format_stamp: "Formatted at 2026-09-26 20:15:04 on dist-test-slave-7034"
server_key: "7db02d3d55e421336c8274a7c29b7be8"
server_key_iv: "b460e6f623cfb20a5f550437d4ccb5e8"
server_key_version: "encryptionkey@0"
I20260926 20:15:04.113449 13746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/wal/instance:
uuid: "3bf95533f9c44c41b59e6fbb35e01e90"
format_stamp: "Formatted at 2026-09-26 20:15:04 on dist-test-slave-7034"
server_key: "7db02d3d55e421336c8274a7c29b7be8"
server_key_iv: "b460e6f623cfb20a5f550437d4ccb5e8"
server_key_version: "encryptionkey@0"
I20260926 20:15:04.117393 13746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 20:15:04.119968 13762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:04.121351 13746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 20:15:04.121502 13746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/wal
uuid: "3bf95533f9c44c41b59e6fbb35e01e90"
format_stamp: "Formatted at 2026-09-26 20:15:04 on dist-test-slave-7034"
server_key: "7db02d3d55e421336c8274a7c29b7be8"
server_key_iv: "b460e6f623cfb20a5f550437d4ccb5e8"
server_key_version: "encryptionkey@0"
I20260926 20:15:04.121623 13746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:04.135066 13746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:04.138556 13746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:04.138769 13746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:04.146267 13746 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.188:33225
I20260926 20:15:04.146565 13814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.188:33225 every 8 connection(s)
I20260926 20:15:04.147372 13746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
I20260926 20:15:04.150328 13815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:04.154206 13815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:15:04.156544 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 13746
I20260926 20:15:04.156683 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790453421225356-23754-0/minicluster-data/master-2/wal/instance
I20260926 20:15:04.156977 23754 external_mini_cluster.cc:1428] Setting key 579a07177fce0b1946a85e8de8b151c2
I20260926 20:15:04.159883 13665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } attempt: 3
Sep 26 20:15:04 dist-test-slave-7034 krb5kdc[13584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.188@KRBTEST.COM
Sep 26 20:15:04 dist-test-slave-7034 krb5kdc[13584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453704, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.188@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
Sep 26 20:15:04 dist-test-slave-7034 krb5kdc[13584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453703, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.188@KRBTEST.COM
I20260926 20:15:04.170606 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.156183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.23.50.190:38339)
0926 20:15:04.156403 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.156407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.157329 (+ 922us) server_negotiation.cc:407] Connection header received
0926 20:15:04.158574 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.158578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.158636 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.158741 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.159947 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.160714 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.161956 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.162239 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.162333 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.166300 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:04.166303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:04.166305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:04.166328 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:04.167949 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.168757 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.168764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.168766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.168849 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.169262 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.169266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.169267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.169653 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:04.169760 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:04.170285 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 20:15:04.170407 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 20:15:04.171002 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.160295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59182)
0926 20:15:04.160467 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:04.160480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:04.160548 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:04.160936 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:04.160939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:04.161137 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:04.161227 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:04.161233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.162268 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.162274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:04.163035 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:04.163051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.163518 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.163521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:04.163621 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.164121 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:04.164135 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:04.166431 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:04.169340 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:04.169345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:04.169348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:04.169663 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:04.170010 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:04.170014 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:04.170016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:04.170100 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:04.170532 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:04.170535 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:04.170617 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 20:15:04.170716 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 20:15:04.170852 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 20:15:04.171890 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.160389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.23.50.188:33225)
0926 20:15:04.160657 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.160665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.160678 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:15:04.160721 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.160724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.160761 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.160889 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.161347 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.162127 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.163157 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.163396 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.163514 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.166581 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:04.166587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:04.166593 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:04.166634 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:04.169180 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.169782 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.169785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.169787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.169912 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.170200 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.170203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.170205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.170417 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:04.171201 (+ 784us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:04.171279 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 20:15:04.171366 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"spinlock_wait_cycles":768,"thread_start_us":121,"threads_started":1}
I20260926 20:15:04.171898 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.157759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.23.50.188:33225)
0926 20:15:04.158201 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.158210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.158245 (+ 35us) server_negotiation.cc:407] Connection header received
0926 20:15:04.158378 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.158393 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.158637 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.158819 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.159797 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.161112 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.162357 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.162732 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.162846 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.165594 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:04.165601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:04.165637 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:04.165669 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:04.168535 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.169095 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.169102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.169104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.169190 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.169513 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.169518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.169520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.170026 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:04.170246 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:04.170941 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 20:15:04.171087 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":344,"spinlock_wait_cycles":384,"thread_start_us":155,"threads_started":1}
I20260926 20:15:04.171898 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.156557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:38339 (local address 127.0.0.1:48488)
0926 20:15:04.157156 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:04.157212 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:04.158378 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:04.158858 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:04.158863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:04.159423 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:04.159797 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:04.159811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.160818 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.160824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:04.161807 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:04.161816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.162493 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.162497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:04.162697 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.163680 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:04.163704 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:04.166098 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:04.168094 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:04.168105 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:04.168116 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:04.168626 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:04.168968 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:04.168973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:04.168978 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:04.169152 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:04.169769 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:04.169775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:04.170026 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 20:15:04.170237 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 20:15:04.170584 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"spinlock_wait_cycles":2816,"thread_start_us":148,"threads_started":1}
I20260926 20:15:04.173527 13815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:15:04.173979 13740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } attempt: 3
I20260926 20:15:04.177659 13665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823: Bootstrap starting.
Sep 26 20:15:04 dist-test-slave-7034 krb5kdc[13584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453703, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.189@KRBTEST.COM for kudu/127.23.50.188@KRBTEST.COM
Sep 26 20:15:04 dist-test-slave-7034 krb5kdc[13584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453704, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.188@KRBTEST.COM for kudu/127.23.50.189@KRBTEST.COM
Sep 26 20:15:04 dist-test-slave-7034 krb5kdc[13584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:15:04.182164 13665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:04.183244 13665 log.cc:824] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:04.185207 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.174470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.23.50.190:38339)
0926 20:15:04.174578 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.174582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.174692 (+ 110us) server_negotiation.cc:407] Connection header received
0926 20:15:04.174831 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.174836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.174885 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.174932 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.175858 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.176571 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.177661 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.178055 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.178143 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.181493 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:04.181497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:04.181499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:04.181526 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:04.183239 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.183762 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.183766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.183768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.183854 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.184240 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.184244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.184246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.184506 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:04.184705 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:04.184912 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 20:15:04.185040 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:15:04.185653 13665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823: No bootstrap required, opened a new log
I20260926 20:15:04.185899 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.174444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59192)
0926 20:15:04.174582 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:04.174599 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:04.174718 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:04.175063 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:04.175066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:04.175272 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:04.175359 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:04.175368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.176545 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.176550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:04.177388 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:04.177399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.178051 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.178054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:04.178139 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.178653 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:04.178672 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:04.181288 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:04.184106 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:04.184109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:04.184111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:04.184375 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:04.184847 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:04.184850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:04.184852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:04.184941 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:04.185366 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:04.185370 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:04.185468 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 20:15:04.185543 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 20:15:04.185717 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 20:15:04.186568 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.174570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.23.50.189:33707)
0926 20:15:04.174661 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.174664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.174917 (+ 253us) server_negotiation.cc:407] Connection header received
0926 20:15:04.175061 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.175064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.175108 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.175161 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.175758 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.176492 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.177526 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.177911 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.177996 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.181008 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:04.181012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:04.181015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:04.181042 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:04.183487 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.184113 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.184117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.184119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.184495 (+ 376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.184942 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.184945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.184947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.185232 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:04.185312 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:04.186316 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0926 20:15:04.186422 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 20:15:04.186570 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.174470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.23.50.188:33225)
0926 20:15:04.174858 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.174863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.174876 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:15:04.174910 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.174913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.174956 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.175466 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.175600 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.176416 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.177535 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.177911 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.178365 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.181441 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:04.181448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:04.181455 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:04.181483 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:04.183981 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.184634 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.184639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.184640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.184739 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.185052 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.185055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.185056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.185238 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:04.186218 (+ 980us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:04.186303 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 20:15:04.186424 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":331}
I20260926 20:15:04.187922 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.174073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:33707 (local address 127.0.0.1:51692)
0926 20:15:04.174857 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:04.174870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:04.174961 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:04.175317 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:04.175319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:04.175491 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:04.175633 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:04.175641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.176612 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.176615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:04.177400 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:04.177408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.178150 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.178154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:04.178289 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.179208 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:04.179226 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:04.180831 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:04.183677 (+ 2846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:04.183683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:04.183686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:04.184012 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:04.184749 (+ 737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:04.184752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:04.184754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:04.184844 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:04.185631 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:04.185633 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:04.185893 (+ 260us) client_negotiation.cc:769] Sending connection context
0926 20:15:04.187608 (+ 1715us) client_negotiation.cc:240] Negotiation successful
0926 20:15:04.187761 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":21}
I20260926 20:15:04.188776 13815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:15:04.189776 13740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82: Bootstrap starting.
I20260926 20:15:04.190199 13665 raft_consensus.cc:348] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } }
I20260926 20:15:04.190398 13665 raft_consensus.cc:729] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06d376c90c44b8b8fed9966f6fd8823, State: Initialized, Role: FOLLOWER
I20260926 20:15:04.190721 13665 consensus_queue.cc:260] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } }
I20260926 20:15:04.192358 13665 sys_catalog.cc:564] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:15:04.192548 13740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82: Neither blocks nor log segments found. Creating new log.
Sep 26 20:15:04 dist-test-slave-7034 krb5kdc[13584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.189@KRBTEST.COM
I20260926 20:15:04.193459 13740 log.cc:824] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:04.195756 13740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82: No bootstrap required, opened a new log
I20260926 20:15:04.197975 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.186947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.23.50.189:33707)
0926 20:15:04.187366 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.187370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.187381 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:15:04.187412 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.187415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.187454 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.187532 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.188322 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.189078 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.190173 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.190480 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.190587 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.193629 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:04.193632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:04.193635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:04.193660 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:04.195665 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.196220 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.196224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.196226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.196314 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.196631 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.196635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.196636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.196857 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:04.197028 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:04.197679 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 20:15:04.197793 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354}
I20260926 20:15:04.198079 13822 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } } }
I20260926 20:15:04.198196 13822 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 20:15:04.198333 13834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:15:04.198377 13834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:15:04.200057 13740 raft_consensus.cc:348] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } }
I20260926 20:15:04.200320 13740 raft_consensus.cc:729] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b82eb7ab3c72402f9abd4a983fe02c82, State: Initialized, Role: FOLLOWER
I20260926 20:15:04.200654 13740 consensus_queue.cc:260] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } }
I20260926 20:15:04.197556 13597 master_runner.cc:186] Error getting master registration for 127.23.50.189:33707: OK, instance_id { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" instance_seqno: 1790453703918570 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:15:04.201745 13597 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:15:04.202069 13740 sys_catalog.cc:564] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 20:15:04 dist-test-slave-7034 krb5kdc[13584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453704, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.188@KRBTEST.COM for kudu/127.23.50.188@KRBTEST.COM
I20260926 20:15:04.205443 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:15:04.206995 13638 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a06d376c90c44b8b8fed9966f6fd8823"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48494
I20260926 20:15:04.207235 13638 raft_consensus.cc:482] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 20:15:04.207366 13638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:04.208403 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.194127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:33225 (local address 127.0.0.1:59196)
0926 20:15:04.194382 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:04.194397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:04.194475 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:04.198912 (+ 4437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:04.198916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:04.199130 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:04.199272 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:04.199279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.200435 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.200440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:04.201286 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:04.201297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.202067 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.202071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:04.202160 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.202793 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:04.202808 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:04.204403 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:04.206580 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:04.206584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:04.206587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:04.206842 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:04.207136 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:04.207138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:04.207140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:04.207236 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:04.207829 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:04.207832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:04.207913 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 20:15:04.207998 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 20:15:04.208170 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260926 20:15:04.209245 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.194788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.23.50.188:33225)
0926 20:15:04.198657 (+ 3869us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.198663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.198681 (+ 18us) server_negotiation.cc:407] Connection header received
0926 20:15:04.198729 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.198732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.198784 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.199381 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.199450 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.200194 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.201494 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.201947 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.204521 (+ 2574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.204645 (+ 124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:04.204651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:04.204656 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:04.204676 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:04.206476 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.206952 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.206955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.206957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.207045 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.207499 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.207502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.207504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.207727 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:04.208883 (+ 1156us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:04.208961 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 20:15:04.209078 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3781}
I20260926 20:15:04.209388 13638 raft_consensus.cc:504] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } }
I20260926 20:15:04.210110 13836 sys_catalog.cc:455] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } } }
I20260926 20:15:04.210283 13836 sys_catalog.cc:458] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:04.210987 13714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a06d376c90c44b8b8fed9966f6fd8823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b82eb7ab3c72402f9abd4a983fe02c82"
I20260926 20:15:04.211233 13714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:04.212366 13815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90: Bootstrap starting.
I20260926 20:15:04.213052 13714 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06d376c90c44b8b8fed9966f6fd8823 in term 1.
I20260926 20:15:04.213486 13616 leader_election.cc:302] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a06d376c90c44b8b8fed9966f6fd8823, b82eb7ab3c72402f9abd4a983fe02c82; no voters:
I20260926 20:15:04.213922 13822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:04.214155 13822 raft_consensus.cc:686] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [term 1 LEADER]: Becoming Leader. State: Replica: a06d376c90c44b8b8fed9966f6fd8823, State: Running, Role: LEADER
W20260926 20:15:04.214156 13847 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:15:04.214313 13847 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:15:04.214479 13822 consensus_queue.cc:237] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } }
I20260926 20:15:04.215238 13815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:04.216117 13815 log.cc:824] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:04.217975 13638 leader_election.cc:288] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [CANDIDATE]: Term 1 election: Requested vote from peers b82eb7ab3c72402f9abd4a983fe02c82 (127.23.50.189:33707), 3bf95533f9c44c41b59e6fbb35e01e90 (127.23.50.188:33225)
I20260926 20:15:04.221522 13849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a06d376c90c44b8b8fed9966f6fd8823. Latest consensus state: current_term: 1 leader_uuid: "a06d376c90c44b8b8fed9966f6fd8823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } } }
I20260926 20:15:04.221670 13849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:04.220996 13672 master_runner.cc:186] Error getting master registration for 127.23.50.188:33225: OK, instance_id { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" instance_seqno: 1790453704144220 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:15:04.224918 13672 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:15:04.225804 13857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:15:04.226938 13857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 20:15:04.233121 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.220244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.23.50.189:33707)
0926 20:15:04.220366 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.220370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.223451 (+ 3081us) server_negotiation.cc:407] Connection header received
0926 20:15:04.223638 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.223642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.223705 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.223757 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.224805 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.225616 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.226919 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.227304 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.227417 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.229317 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:04.229321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:04.229324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:04.229354 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:04.231137 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.231652 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.231656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.231658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.231747 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.232065 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.232071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.232074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.232300 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:04.232402 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:04.232780 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 20:15:04.232910 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:15:04.236227 13815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90: No bootstrap required, opened a new log
I20260926 20:15:04.239616 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.222150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.23.50.188:33225)
0926 20:15:04.222263 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.222267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.227306 (+ 5039us) server_negotiation.cc:407] Connection header received
0926 20:15:04.227510 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.227513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.227567 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.227642 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.229312 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.230143 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.232099 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.232427 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.232503 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.235140 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:04.235144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:04.235146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:04.235172 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:04.237006 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.237767 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.237784 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.237786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.237922 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.238384 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.238388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.238401 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.238635 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:04.238751 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:04.239306 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 20:15:04.239432 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 20:15:04.241863 13815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } }
I20260926 20:15:04.242158 13815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bf95533f9c44c41b59e6fbb35e01e90, State: Initialized, Role: FOLLOWER
I20260926 20:15:04.242542 13815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } }
I20260926 20:15:04.243148 13861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } } }
I20260926 20:15:04.243345 13861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:04.244081 13815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:15:04.244159 13789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a06d376c90c44b8b8fed9966f6fd8823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3bf95533f9c44c41b59e6fbb35e01e90"
I20260926 20:15:04.244388 13789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:04.246102 13789 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06d376c90c44b8b8fed9966f6fd8823 in term 1.
I20260926 20:15:04.246553 13714 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [term 1 FOLLOWER]: Refusing update from remote peer a06d376c90c44b8b8fed9966f6fd8823: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:04.247047 13788 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [term 1 FOLLOWER]: Refusing update from remote peer a06d376c90c44b8b8fed9966f6fd8823: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:04.247298 13849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:04.248122 13849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:04.251459 13836 sys_catalog.cc:455] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a06d376c90c44b8b8fed9966f6fd8823. Latest consensus state: current_term: 1 leader_uuid: "a06d376c90c44b8b8fed9966f6fd8823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } } }
I20260926 20:15:04.251600 13836 sys_catalog.cc:458] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:04.256016 13849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a06d376c90c44b8b8fed9966f6fd8823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } } }
I20260926 20:15:04.256158 13849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:04.256315 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:04.222889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.23.50.190:38339)
0926 20:15:04.223004 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.223009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.233930 (+ 10921us) server_negotiation.cc:407] Connection header received
0926 20:15:04.246425 (+ 12495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.246430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.246483 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.246535 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.247590 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.248323 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.249426 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:04.249738 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:04.249817 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:04.252459 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:04.252464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:04.252467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:04.252495 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:04.254330 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.254848 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.254852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.254853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.254950 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:04.255286 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:04.255289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:04.255291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:04.255508 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:04.255610 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:04.255983 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 20:15:04.256120 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 20:15:04.258039 13878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a06d376c90c44b8b8fed9966f6fd8823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } } }
I20260926 20:15:04.258172 13878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:04.252722 13861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a06d376c90c44b8b8fed9966f6fd8823. Latest consensus state: current_term: 1 leader_uuid: "a06d376c90c44b8b8fed9966f6fd8823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } } }
I20260926 20:15:04.258479 13861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:04.268091 13822 sys_catalog.cc:455] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a06d376c90c44b8b8fed9966f6fd8823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } } }
I20260926 20:15:04.268244 13822 sys_catalog.cc:458] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823 [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:04.269384 13836 sys_catalog.cc:455] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a06d376c90c44b8b8fed9966f6fd8823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a06d376c90c44b8b8fed9966f6fd8823" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 38339 } } peers { permanent_uuid: "b82eb7ab3c72402f9abd4a983fe02c82" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 33707 } } peers { permanent_uuid: "3bf95533f9c44c41b59e6fbb35e01e90" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 33225 } } }
I20260926 20:15:04.269513 13836 sys_catalog.cc:458] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:04.270020 13867 mvcc.cc:205] Tried to move back new op lower bound from 7333698372582932480 to 7333698372466077696. Current Snapshot: MvccSnapshot[applied={T|T < 7333698372582932480}]
I20260926 20:15:04.271551 13876 catalog_manager.cc:1259] Loaded cluster ID: da4b2eabb6bf4ae38492baf6104b6ef6
I20260926 20:15:04.271618 13876 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90: loading cluster ID for follower catalog manager: success
I20260926 20:15:04.271641 13857 catalog_manager.cc:1347] Generated new cluster ID: da4b2eabb6bf4ae38492baf6104b6ef6
I20260926 20:15:04.271696 13857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
W20260926 20:15:04.272140 13876 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260926 20:15:04.272235 13876 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:15:04.296458 13857 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:15:04.297786 13857 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:15:04.319227 13857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a06d376c90c44b8b8fed9966f6fd8823: Generated new TSK 0
I20260926 20:15:04.320192 13857 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:15:04.358464 13743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.355433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.23.50.189:33707)
0926 20:15:04.355571 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.355576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.356993 (+ 1417us) server_negotiation.cc:407] Connection header received
0926 20:15:04.357210 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.357214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.357269 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.357322 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.358315 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51728: BlockingRecv error: recv got EOF from 127.0.0.1:51728 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 20:15:04.359935 13667 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.356312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48508 (local address 127.23.50.190:38339)
0926 20:15:04.356431 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.356435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.358086 (+ 1651us) server_negotiation.cc:407] Connection header received
0926 20:15:04.359078 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.359082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.359140 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.359217 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.359815 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48508: BlockingRecv error: recv got EOF from 127.0.0.1:48508 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 20:15:04.361820 13820 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.360457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.23.50.188:33225)
0926 20:15:04.360591 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.360596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.360931 (+ 335us) server_negotiation.cc:407] Connection header received
0926 20:15:04.361090 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.361093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.361139 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.361188 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.361689 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59210: BlockingRecv error: recv got EOF from 127.0.0.1:59210 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 20:15:04.366556 13743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.364816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51738 (local address 127.23.50.189:33707)
0926 20:15:04.364929 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.364933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.365461 (+ 528us) server_negotiation.cc:407] Connection header received
0926 20:15:04.365694 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.365698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.365774 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.365829 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.366455 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51738: BlockingRecv error: recv got EOF from 127.0.0.1:51738 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 20:15:04.371606 13743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.370034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.23.50.189:33707)
0926 20:15:04.370138 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.370142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.370714 (+ 572us) server_negotiation.cc:407] Connection header received
0926 20:15:04.370901 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.370904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.370953 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.371002 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.371501 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51750: BlockingRecv error: recv got EOF from 127.0.0.1:51750 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 20:15:04.377379 13743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.375691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.23.50.189:33707)
0926 20:15:04.375795 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.375800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.376367 (+ 567us) server_negotiation.cc:407] Connection header received
0926 20:15:04.376570 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.376574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.376643 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.376709 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.377270 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51766: BlockingRecv error: recv got EOF from 127.0.0.1:51766 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 20:15:04.385507 13743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.383695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.23.50.189:33707)
0926 20:15:04.383861 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.383866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.384475 (+ 609us) server_negotiation.cc:407] Connection header received
0926 20:15:04.384713 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.384717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.384772 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.384830 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.385391 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51774: BlockingRecv error: recv got EOF from 127.0.0.1:51774 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 20:15:04.398010 13743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.396085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.23.50.189:33707)
0926 20:15:04.396212 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.396217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.396855 (+ 638us) server_negotiation.cc:407] Connection header received
0926 20:15:04.397086 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.397090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.397150 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.397210 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.397835 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51780: BlockingRecv error: recv got EOF from 127.0.0.1:51780 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 20:15:04.418229 13743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.416532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.23.50.189:33707)
0926 20:15:04.416663 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.416668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.417233 (+ 565us) server_negotiation.cc:407] Connection header received
0926 20:15:04.417427 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.417431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.417482 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.417533 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.418107 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51796: BlockingRecv error: recv got EOF from 127.0.0.1:51796 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 20:15:04.454247 13743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.452915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.23.50.189:33707)
0926 20:15:04.453062 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.453068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.453272 (+ 204us) server_negotiation.cc:407] Connection header received
0926 20:15:04.453371 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.453374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.453436 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.453552 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.454123 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51808: BlockingRecv error: recv got EOF from 127.0.0.1:51808 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 20:15:04.521571 13743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.520452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51824 (local address 127.23.50.189:33707)
0926 20:15:04.520616 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.520620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.520741 (+ 121us) server_negotiation.cc:407] Connection header received
0926 20:15:04.520864 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.520867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.520917 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.521001 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.521473 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51824: BlockingRecv error: recv got EOF from 127.0.0.1:51824 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 20:15:04.653257 13743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.651943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.23.50.189:33707)
0926 20:15:04.652114 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.652119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.652313 (+ 194us) server_negotiation.cc:407] Connection header received
0926 20:15:04.652503 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.652505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.652554 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.652655 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.653116 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51832: BlockingRecv error: recv got EOF from 127.0.0.1:51832 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 20:15:04.912531 13743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:04.911451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.23.50.189:33707)
0926 20:15:04.911642 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:04.911646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:04.911662 (+ 16us) server_negotiation.cc:407] Connection header received
0926 20:15:04.911773 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:04.911775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:04.911825 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:04.911900 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:04.912412 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51844: BlockingRecv error: recv got EOF from 127.0.0.1:51844 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 20:15:05.215687 13847 catalog_manager.cc:1259] Loaded cluster ID: da4b2eabb6bf4ae38492baf6104b6ef6
I20260926 20:15:05.215776 13847 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82: loading cluster ID for follower catalog manager: success
I20260926 20:15:05.217442 13847 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82: acquiring CA information for follower catalog manager: success
I20260926 20:15:05.218645 13847 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b82eb7ab3c72402f9abd4a983fe02c82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:15:05.274677 13876 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90: acquiring CA information for follower catalog manager: success
I20260926 20:15:05.275959 13876 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bf95533f9c44c41b59e6fbb35e01e90: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:15:05.430761 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:05.426691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.23.50.189:33707)
0926 20:15:05.426944 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:05.426948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:05.427062 (+ 114us) server_negotiation.cc:407] Connection header received
0926 20:15:05.427244 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:05.427252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:05.427384 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:05.427506 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:05.427771 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:05.428327 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:05.429222 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:05.429456 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:05.429570 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:05.429922 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:05.430359 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:05.430506 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:05.430543 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 20:15:05.430599 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260926 20:15:05.437580 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:05.433882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59218 (local address 127.23.50.188:33225)
0926 20:15:05.434131 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:05.434135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:05.434150 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:15:05.434274 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:05.434277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:05.434393 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:05.434521 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:05.434736 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:05.435272 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:05.436101 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:05.436333 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:05.436449 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:05.436748 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:05.437164 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:05.437305 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:05.437368 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 20:15:05.437423 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
I20260926 20:15:05.444803 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:05.440611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.23.50.189:33707)
0926 20:15:05.440753 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:05.440758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:05.441266 (+ 508us) server_negotiation.cc:407] Connection header received
0926 20:15:05.441425 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:05.441429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:05.441491 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:05.441563 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:05.441960 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:05.442710 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:05.443575 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:05.443863 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:05.443982 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:05.444263 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:05.444366 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:05.444429 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:05.444593 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 20:15:05.444647 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 20:15:05.445511 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:05.440894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59232 (local address 127.23.50.188:33225)
0926 20:15:05.441016 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:05.441023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:05.441177 (+ 154us) server_negotiation.cc:407] Connection header received
0926 20:15:05.441355 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:05.441358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:05.441414 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:05.441477 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:05.441889 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:05.442673 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:05.443942 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:05.444306 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:05.444425 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:05.444925 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:05.445130 (+ 205us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:05.445232 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:05.445327 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 20:15:05.445379 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 20:15:05.445806 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:05.440582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.23.50.190:38339)
0926 20:15:05.440887 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:05.440892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:05.441098 (+ 206us) server_negotiation.cc:407] Connection header received
0926 20:15:05.441338 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:05.441342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:05.441505 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:05.441640 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:05.442031 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:05.442549 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:05.443906 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:05.444281 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:05.444400 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:05.444764 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:05.445189 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:05.445271 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:05.445575 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 20:15:05.445640 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260926 20:15:05.449075 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 13597
I20260926 20:15:05.455950 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 13672
I20260926 20:15:05.461957 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 13746
2026-09-26T20:15:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3164 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 20:15:05 dist-test-slave-7034 krb5kdc[13962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 20:15:05 dist-test-slave-7034 krb5kdc[13962](info): set up 2 sockets
Sep 26 20:15:05 dist-test-slave-7034 krb5kdc[13962](info): commencing operation
krb5kdc: starting...
W20260926 20:15:06.514900 23754 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/test-user.keytab.
Sep 26 20:15:06 dist-test-slave-7034 krb5kdc[13962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T20:15:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T20:15:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.50.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
Entry for principal kudu/127.23.50.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab.
I20260926 20:15:06.641638 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:43967
--webserver_interface=127.23.50.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:43967,127.23.50.189:35793,127.23.50.188:38747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 20:15:06.755599 13975 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:15:06.755926 13975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:06.755951 13975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:06.756048 13975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:06.760365 13975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:06.760432 13975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:06.760450 13975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:06.760468 13975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:06.760526 13975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:06.766108 13975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:43967,127.23.50.189:35793,127.23.50.188:38747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.190.keytab
--principal=kudu/127.23.50.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.190:43967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.50.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:06.767351 13975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:06.768497 13975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:06.777508 13980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:06.777499 13983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:06.777525 13981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:06.777858 13975 server_base.cc:1034] running on GCE node
I20260926 20:15:06.778599 13975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:06.779760 13975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:06.781034 13975 hybrid_clock.cc:648] HybridClock initialized: now 1790453706781016 us; error 28 us; skew 500 ppm
Sep 26 20:15:06 dist-test-slave-7034 krb5kdc[13962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453706, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:06.790627 13975 init.cc:377] Logged in from keytab as kudu/127.23.50.190@KRBTEST.COM (short username kudu)
I20260926 20:15:06.791714 13975 webserver.cc:460] Webserver started at http://127.23.50.190:39327/ using document root <none> and password file <none>
I20260926 20:15:06.792337 13975 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:06.792434 13975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:06.792682 13975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:06.794914 13975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/data/instance:
uuid: "b57d84be20c841fab3f3b860725fa5ce"
format_stamp: "Formatted at 2026-09-26 20:15:06 on dist-test-slave-7034"
server_key: "a6e996583c4bb451f4c251f32a451a1e"
server_key_iv: "eac72d93c4b30e94d01c2cf234fb5949"
server_key_version: "encryptionkey@0"
I20260926 20:15:06.795508 13975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/wal/instance:
uuid: "b57d84be20c841fab3f3b860725fa5ce"
format_stamp: "Formatted at 2026-09-26 20:15:06 on dist-test-slave-7034"
server_key: "a6e996583c4bb451f4c251f32a451a1e"
server_key_iv: "eac72d93c4b30e94d01c2cf234fb5949"
server_key_version: "encryptionkey@0"
I20260926 20:15:06.799374 13975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 20:15:06.801981 13990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:06.803246 13975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:15:06.803417 13975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/wal
uuid: "b57d84be20c841fab3f3b860725fa5ce"
format_stamp: "Formatted at 2026-09-26 20:15:06 on dist-test-slave-7034"
server_key: "a6e996583c4bb451f4c251f32a451a1e"
server_key_iv: "eac72d93c4b30e94d01c2cf234fb5949"
server_key_version: "encryptionkey@0"
I20260926 20:15:06.803537 13975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:06.825457 13975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:06.828475 13975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:06.828686 13975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:06.835528 13975 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.190:43967
I20260926 20:15:06.835834 14042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.190:43967 every 8 connection(s)
I20260926 20:15:06.836534 13975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/data/info.pb
I20260926 20:15:06.838518 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 13975
I20260926 20:15:06.838665 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-0/wal/instance
I20260926 20:15:06.838984 23754 external_mini_cluster.cc:1428] Setting key 8cc3bc7216619e7bdee87bd9006f3034
I20260926 20:15:06.839553 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:06.843802 14043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.50.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.189@KRBTEST.COM" created.
Sep 26 20:15:06 dist-test-slave-7034 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453706, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:15:06.862285 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:06.846208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.23.50.190:43967)
0926 20:15:06.846632 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:06.846651 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:06.846978 (+ 327us) server_negotiation.cc:407] Connection header received
0926 20:15:06.848242 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:06.848247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:06.848510 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:06.848691 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:06.850095 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:06.851741 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:06.853603 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:06.854250 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:06.854429 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:06.857546 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:06.857552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:06.857562 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:06.857584 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:06.859298 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:06.859763 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:06.859768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:06.859770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:06.859871 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:06.860176 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:06.860178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:06.860179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:06.860636 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:06.860741 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:06.861124 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 20:15:06.861515 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":269,"thread_start_us":142,"threads_started":1}
I20260926 20:15:06.862303 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:06.846209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43967 (local address 127.0.0.1:48152)
0926 20:15:06.846780 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:06.846816 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:06.847732 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:06.848872 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:06.848909 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:06.849476 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:06.849835 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:06.849912 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:06.852096 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:06.852099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:06.853447 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:06.853459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:06.854465 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:06.854470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:06.854667 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:06.855381 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:06.855409 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:06.857367 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:06.859402 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:06.859411 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:06.859420 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:06.859641 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:06.859953 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:06.859956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:06.859957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:06.860068 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:06.860733 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:06.860738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:06.860867 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 20:15:06.861012 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 20:15:06.861421 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":66,"thread_start_us":154,"threads_started":1}
I20260926 20:15:06.864818 14043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:15:06.865563 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:06.865311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:35793 (local address 127.0.0.1:33364)
0926 20:15:06.865435 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:06.865505 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 20:15:06.865796 13994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.50.189:35793: connect: Connection refused (error 111)
I20260926 20:15:06.866689 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:06.866476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:35793 (local address 127.0.0.1:33368)
0926 20:15:06.866589 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:06.866640 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 20:15:06.866837 14043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.189:35793: Network error: Client connection negotiation failed: client connection to 127.23.50.189:35793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.189.keytab.
Entry for principal kudu/127.23.50.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.189.keytab.
I20260926 20:15:06.869628 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.189:35793
--webserver_interface=127.23.50.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.189.keytab
--principal=kudu/127.23.50.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:43967,127.23.50.189:35793,127.23.50.188:38747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 20:15:06.916095 14043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } attempt: 1
I20260926 20:15:06.917039 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:06.916732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:35793 (local address 127.0.0.1:33372)
0926 20:15:06.916892 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:06.916960 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 20:15:06.917789 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:06.917579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:35793 (local address 127.0.0.1:33380)
0926 20:15:06.917690 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:06.917737 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:15:06.917956 14043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.189:35793: Network error: Client connection negotiation failed: client connection to 127.23.50.189:35793: connect: Connection refused (error 111)
W20260926 20:15:06.980953 14050 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:15:06.981251 14050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:06.981310 14050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:06.981390 14050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 20:15:06.985507 14050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:06.985569 14050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:06.985589 14050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:06.985647 14050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:06.985680 14050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:06.986124 14043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } attempt: 2
I20260926 20:15:06.987001 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:06.986703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:35793 (local address 127.0.0.1:33394)
0926 20:15:06.986863 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:06.986926 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 20:15:06.987743 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:06.987553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:35793 (local address 127.0.0.1:33410)
0926 20:15:06.987649 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:06.987693 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.189:35793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:15:06.987951 14043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.189:35793: Network error: Client connection negotiation failed: client connection to 127.23.50.189:35793: connect: Connection refused (error 111)
I20260926 20:15:06.991758 14050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:43967,127.23.50.189:35793,127.23.50.188:38747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.189.keytab
--principal=kudu/127.23.50.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.189:35793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.50.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:06.992852 14050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:06.993924 14050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 20:15:07.002620 14058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:07.002635 14055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:07.002682 14056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:07.002729 14050 server_base.cc:1034] running on GCE node
I20260926 20:15:07.003470 14050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:07.004678 14050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:07.005929 14050 hybrid_clock.cc:648] HybridClock initialized: now 1790453707005904 us; error 38 us; skew 500 ppm
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:07.009051 14050 init.cc:377] Logged in from keytab as kudu/127.23.50.189@KRBTEST.COM (short username kudu)
I20260926 20:15:07.009984 14050 webserver.cc:460] Webserver started at http://127.23.50.189:40389/ using document root <none> and password file <none>
I20260926 20:15:07.010515 14050 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:07.010574 14050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:07.010819 14050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:07.012850 14050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/data/instance:
uuid: "f08e5d792cc34394ba39241b8a32c02e"
format_stamp: "Formatted at 2026-09-26 20:15:07 on dist-test-slave-7034"
server_key: "090ac22dcf5140982897e319b7a114ca"
server_key_iv: "05a6a5e456dfdae94b04bc476f7577f8"
server_key_version: "encryptionkey@0"
I20260926 20:15:07.013401 14050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/wal/instance:
uuid: "f08e5d792cc34394ba39241b8a32c02e"
format_stamp: "Formatted at 2026-09-26 20:15:07 on dist-test-slave-7034"
server_key: "090ac22dcf5140982897e319b7a114ca"
server_key_iv: "05a6a5e456dfdae94b04bc476f7577f8"
server_key_version: "encryptionkey@0"
I20260926 20:15:07.017431 14050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 20:15:07.020066 14065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:07.021415 14050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 20:15:07.021565 14050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/wal
uuid: "f08e5d792cc34394ba39241b8a32c02e"
format_stamp: "Formatted at 2026-09-26 20:15:07 on dist-test-slave-7034"
server_key: "090ac22dcf5140982897e319b7a114ca"
server_key_iv: "05a6a5e456dfdae94b04bc476f7577f8"
server_key_version: "encryptionkey@0"
I20260926 20:15:07.021683 14050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:07.042150 14050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:07.045574 14050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:07.045795 14050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:07.053212 14050 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.189:35793
I20260926 20:15:07.053576 14117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.189:35793 every 8 connection(s)
I20260926 20:15:07.054446 14050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/data/info.pb
I20260926 20:15:07.055847 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 14050
I20260926 20:15:07.055994 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-1/wal/instance
I20260926 20:15:07.056303 23754 external_mini_cluster.cc:1428] Setting key 2320e807e57b6ab202bdc9339d8b3ee0
I20260926 20:15:07.060680 14118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
WARNING: no policy specified for kudu/127.23.50.188@KRBTEST.COM; defaulting to no policy
I20260926 20:15:07.064682 14118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.50.188@KRBTEST.COM" created.
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.189@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:15:07.079159 14043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } attempt: 3
I20260926 20:15:07.080119 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.066762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.23.50.190:43967)
0926 20:15:07.066930 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.066935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.068105 (+ 1170us) server_negotiation.cc:407] Connection header received
0926 20:15:07.069109 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.069113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.069183 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.069280 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.070580 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.071107 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.072263 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.072503 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.072610 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.075399 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.075402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.075404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.075430 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.077561 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.078212 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.078220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.078221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.078301 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.078666 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.078670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.078672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.078922 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.079044 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.079759 (+ 715us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.079896 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 20:15:07.081112 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.067272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43967 (local address 127.0.0.1:48164)
0926 20:15:07.067938 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.067984 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:07.068885 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:07.069458 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:07.069464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:07.069900 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:07.070434 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.070448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.071252 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.071255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.072130 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.072139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.072660 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.072664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.072834 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.073469 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:07.073485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:07.075215 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:07.077698 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.077704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.077721 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.078115 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.078397 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.078401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.078403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.078554 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.079045 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:07.079053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:07.079435 (+ 382us) client_negotiation.cc:769] Sending connection context
0926 20:15:07.079668 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 20:15:07.080036 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":160,"threads_started":1}
I20260926 20:15:07.082793 14118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.50.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.188.keytab.
Entry for principal kudu/127.23.50.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.188.keytab.
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453706, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.189@KRBTEST.COM
I20260926 20:15:07.086627 23754 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
/tmp/dist-test-taskFqzGjO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.188:38747
--webserver_interface=127.23.50.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.188.keytab
--principal=kudu/127.23.50.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.50.148:46451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.50.190:43967,127.23.50.189:35793,127.23.50.188:38747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.189@KRBTEST.COM for kudu/127.23.50.189@KRBTEST.COM
I20260926 20:15:07.090554 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.079700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:35793 (local address 127.0.0.1:33416)
0926 20:15:07.079899 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.079914 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:07.080007 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:07.080749 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:07.080753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:07.080923 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:07.081022 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.081028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.082360 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.082363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.083085 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.083095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.083917 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.083920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.084049 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.084632 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:07.084650 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:07.086243 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:07.088647 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.088651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.088653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.088859 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.089186 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.089190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.089192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.089276 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.089903 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:07.089907 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:07.090038 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 20:15:07.090185 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 20:15:07.090357 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260926 20:15:07.090813 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.079803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.23.50.189:35793)
0926 20:15:07.080357 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.080376 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.080396 (+ 20us) server_negotiation.cc:407] Connection header received
0926 20:15:07.080432 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.080436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.080613 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.080729 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.081141 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.082228 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.083207 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.083817 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.083916 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.086409 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.086420 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.086436 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.086467 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.088533 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.088977 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.088984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.088989 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.089082 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.089377 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.089382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.089384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.089707 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.089894 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.090317 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.090475 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":166,"threads_started":1}
I20260926 20:15:07.093011 14043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:15:07.093780 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.093525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:32920)
0926 20:15:07.093657 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.093714 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 20:15:07.094568 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.094310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:32928)
0926 20:15:07.094456 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.094514 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260926 20:15:07.094782 14043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:38747: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
I20260926 20:15:07.095204 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.083442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.23.50.189:35793)
0926 20:15:07.083782 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.083793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.083810 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:15:07.083852 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.083857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.083904 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.084025 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.084602 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.085420 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.086520 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.086935 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.087064 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.089458 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.089461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.089464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.089485 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.093243 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.093760 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.093765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.093768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.093892 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.094204 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.094209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.094212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.094435 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.094540 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.094948 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.095052 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":156,"threads_started":1}
I20260926 20:15:07.095234 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.083416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:35793 (local address 127.0.0.1:33418)
0926 20:15:07.083546 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.083561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:07.083641 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:07.084035 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:07.084038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:07.084276 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:07.084431 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.084444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.085544 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.085550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.086412 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.086421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.087361 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.087364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.087428 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.087783 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:07.087800 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:07.089338 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:07.093372 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.093376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.093381 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.093647 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.093999 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.094004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.094009 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.094101 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.094581 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:07.094586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:07.094682 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 20:15:07.094816 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 20:15:07.094974 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 20:15:07.096036 14118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:15:07.096743 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.096486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:32930)
0926 20:15:07.096620 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.096690 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 20:15:07.096983 14069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
I20260926 20:15:07.097913 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.097662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:32942)
0926 20:15:07.097771 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.097814 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 20:15:07.098093 14118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:38747: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
I20260926 20:15:07.125914 14043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } attempt: 1
I20260926 20:15:07.126729 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.126438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:32956)
0926 20:15:07.126586 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.126651 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 20:15:07.127434 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.127205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:32966)
0926 20:15:07.127340 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.127381 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 20:15:07.127559 14043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:38747: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
I20260926 20:15:07.147380 14118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } attempt: 1
I20260926 20:15:07.148260 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.147961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:32970)
0926 20:15:07.148118 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.148181 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 20:15:07.148947 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.148736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:32986)
0926 20:15:07.148842 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.148893 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 20:15:07.149078 14118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:38747: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
W20260926 20:15:07.200862 14125 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 20:15:07.201207 14125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 20:15:07.201289 14125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 20:15:07.201373 14125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 20:15:07.202708 14043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } attempt: 2
I20260926 20:15:07.203495 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.203204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:32998)
0926 20:15:07.203345 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.203420 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 20:15:07.204198 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.203981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:33012)
0926 20:15:07.204090 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.204137 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 20:15:07.204331 14043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:38747: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
W20260926 20:15:07.205793 14125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 20:15:07.205891 14125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 20:15:07.205960 14125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 20:15:07.205997 14125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 20:15:07.206033 14125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 20:15:07.211602 14125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.50.148:46451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.50.190:43967,127.23.50.189:35793,127.23.50.188:38747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/krb5kdc/kudu_127.23.50.188.keytab
--principal=kudu/127.23.50.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.50.188:38747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.50.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 19:25:01 UTC on ccycloud.quasar-kzaivi.root.comops.site
build id 4675015
I20260926 20:15:07.212697 14125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 20:15:07.213792 14125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 20:15:07.217239 14118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } attempt: 2
I20260926 20:15:07.218080 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.217749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:33016)
0926 20:15:07.217927 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.217996 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 20:15:07.219034 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.218790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:33032)
0926 20:15:07.218898 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.218945 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 20:15:07.219228 14118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.50.188:38747: Network error: Client connection negotiation failed: client connection to 127.23.50.188:38747: connect: Connection refused (error 111)
W20260926 20:15:07.222482 14131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:07.222558 14125 server_base.cc:1034] running on GCE node
W20260926 20:15:07.222529 14134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 20:15:07.222718 14132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 20:15:07.223332 14125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 20:15:07.224421 14125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 20:15:07.225582 14125 hybrid_clock.cc:648] HybridClock initialized: now 1790453707225570 us; error 42 us; skew 500 ppm
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 20:15:07.228425 14125 init.cc:377] Logged in from keytab as kudu/127.23.50.188@KRBTEST.COM (short username kudu)
I20260926 20:15:07.229249 14125 webserver.cc:460] Webserver started at http://127.23.50.188:40581/ using document root <none> and password file <none>
I20260926 20:15:07.229760 14125 fs_manager.cc:359] Metadata directory not provided
I20260926 20:15:07.229815 14125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 20:15:07.230053 14125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 20:15:07.232124 14125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/data/instance:
uuid: "a5eb4c1f5cae4697bf353613264fdc83"
format_stamp: "Formatted at 2026-09-26 20:15:07 on dist-test-slave-7034"
server_key: "7d657e43701675f15f1ef6eb906ce618"
server_key_iv: "5d5aac26c42cf59398085b81788dad75"
server_key_version: "encryptionkey@0"
I20260926 20:15:07.232638 14125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/wal/instance:
uuid: "a5eb4c1f5cae4697bf353613264fdc83"
format_stamp: "Formatted at 2026-09-26 20:15:07 on dist-test-slave-7034"
server_key: "7d657e43701675f15f1ef6eb906ce618"
server_key_iv: "5d5aac26c42cf59398085b81788dad75"
server_key_version: "encryptionkey@0"
I20260926 20:15:07.236277 14125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 20:15:07.238696 14141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 20:15:07.239969 14125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 20:15:07.240110 14125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/data,/tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/wal
uuid: "a5eb4c1f5cae4697bf353613264fdc83"
format_stamp: "Formatted at 2026-09-26 20:15:07 on dist-test-slave-7034"
server_key: "7d657e43701675f15f1ef6eb906ce618"
server_key_iv: "5d5aac26c42cf59398085b81788dad75"
server_key_version: "encryptionkey@0"
I20260926 20:15:07.240257 14125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 20:15:07.261894 14125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 20:15:07.265260 14125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 20:15:07.265475 14125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 20:15:07.272341 14125 rpc_server.cc:307] RPC server started. Bound to: 127.23.50.188:38747
I20260926 20:15:07.272637 14193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.50.188:38747 every 8 connection(s)
I20260926 20:15:07.273317 14125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/data/info.pb
I20260926 20:15:07.274544 23754 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu as pid 14125
I20260926 20:15:07.274686 23754 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFqzGjO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790453421225356-23754-0/minicluster-data/master-2/wal/instance
I20260926 20:15:07.274981 23754 external_mini_cluster.cc:1428] Setting key 574f54695a3c5fdb7534dcc1ba46cc32
I20260926 20:15:07.276331 14194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 20:15:07.280120 14194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.188@KRBTEST.COM
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.188@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
I20260926 20:15:07.289553 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.276333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.23.50.188:38747)
0926 20:15:07.276750 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.276766 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.276803 (+ 37us) server_negotiation.cc:407] Connection header received
0926 20:15:07.277449 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.277465 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.277822 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.278244 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.279061 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.280054 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.280969 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.281463 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.281630 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.284552 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.284557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.284566 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.284593 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.286542 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.287006 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.287033 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.287039 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.287208 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.287519 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.287526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.287528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.287925 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.288178 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.288475 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.288801 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":135,"threads_started":1}
I20260926 20:15:07.291116 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.281887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.190:43967 (local address 127.0.0.1:48168)
0926 20:15:07.282299 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.282347 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:07.282464 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:07.282751 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:07.282755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:07.283078 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:07.283199 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.283204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.284071 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.284078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.284887 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.284897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.285397 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.285401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.285513 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.285974 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:07.285986 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:07.287581 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:07.289407 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.289418 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.289424 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.289643 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.290067 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.290070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.290072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.290158 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.290557 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:07.290563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:07.290686 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 20:15:07.290831 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 20:15:07.290981 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":122,"threads_started":1}
I20260926 20:15:07.291283 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.281905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.23.50.190:43967)
0926 20:15:07.282060 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.282065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.282447 (+ 382us) server_negotiation.cc:407] Connection header received
0926 20:15:07.282548 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.282551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.282609 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.282704 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.283332 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.283942 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.285005 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.285274 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.285397 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.287742 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.287746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.287748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.287771 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.289291 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.289823 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.289829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.289832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.289951 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.290254 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.290257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.290258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.290448 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.290529 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.291010 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.291132 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 20:15:07.292316 14194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.190@KRBTEST.COM
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.188@KRBTEST.COM for kudu/127.23.50.189@KRBTEST.COM
I20260926 20:15:07.301611 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.292205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48172 (local address 127.23.50.190:43967)
0926 20:15:07.292306 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.292309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.292323 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:15:07.292432 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.292436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.292479 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.292558 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.293371 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.293894 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.294885 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.295115 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.295216 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.298038 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.298042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.298045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.298074 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.299778 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.300296 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.300299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.300300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.300377 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.300676 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.300681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.300683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.300863 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.301096 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.301218 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.301341 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 20:15:07.302002 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.292792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.189:35793 (local address 127.0.0.1:33422)
0926 20:15:07.292913 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.292928 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:07.292998 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:07.293947 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:07.293953 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:07.294114 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:07.294208 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.294212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.295134 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.295139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.295891 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.295901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.296597 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.296599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.296671 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.297062 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:07.297099 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:07.298577 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:07.300441 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.300445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.300447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.300642 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.300932 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.300935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.300936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.301020 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.301482 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:07.301485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:07.301571 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 20:15:07.301652 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 20:15:07.301823 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 20:15:07.302085 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.292995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.23.50.189:35793)
0926 20:15:07.293144 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.293148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.293163 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:15:07.293214 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.293217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.293261 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.293351 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.294333 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.295035 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.296012 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.296475 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.296605 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.298748 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.298751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.298753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.298778 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.300337 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.300747 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.300751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.300753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.300831 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.301117 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.301120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.301122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.301339 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.301435 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.301754 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.301928 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 20:15:07.302809 14194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } has no permanent_uuid. Determining permanent_uuid...
I20260926 20:15:07.303481 14043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } attempt: 3
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.188@KRBTEST.COM for kudu/127.23.50.188@KRBTEST.COM
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.50.189@KRBTEST.COM
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453706, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.190@KRBTEST.COM for kudu/127.23.50.188@KRBTEST.COM
I20260926 20:15:07.310386 14118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } attempt: 3
I20260926 20:15:07.312804 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.303268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:33044)
0926 20:15:07.303387 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.303403 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:07.303473 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:07.303683 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:07.303688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:07.304151 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:07.304242 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.304249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.305265 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.305268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.306053 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.306060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.306688 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.306691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.306771 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.307208 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:07.307235 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:07.309052 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:07.311088 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.311093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.311096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.311405 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.311692 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.311695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.311697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.311776 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.312247 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:07.312251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:07.312333 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 20:15:07.312443 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 20:15:07.312593 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 20:15:07.313090 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.303323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.23.50.188:38747)
0926 20:15:07.303434 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.303476 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.303491 (+ 15us) server_negotiation.cc:407] Connection header received
0926 20:15:07.303549 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.303551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.303595 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.303669 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.304367 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.305148 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.306130 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.306455 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.306567 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.309185 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.309188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.309191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.309213 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.310971 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.311519 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.311521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.311524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.311601 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.311908 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.311912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.311913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.312127 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.312211 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.312508 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.312665 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":48}
I20260926 20:15:07.313951 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.303823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:33052)
0926 20:15:07.303918 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.303930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:07.303999 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:07.304335 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:07.304338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:07.304507 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:07.304591 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.304597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.306064 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.306068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.306836 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.306847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.307596 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.307599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.307684 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.308156 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:07.308176 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:07.310178 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:07.312164 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.312170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.312172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.312439 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.312738 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.312740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.312742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.312828 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.313374 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:07.313380 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:07.313489 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 20:15:07.313580 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 20:15:07.313720 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 20:15:07.314610 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.303856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.23.50.188:38747)
0926 20:15:07.304159 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.304168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.304177 (+ 9us) server_negotiation.cc:407] Connection header received
0926 20:15:07.304201 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.304204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.304232 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.304321 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.304709 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.305956 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.307240 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.307478 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.307592 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.310349 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.310353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.310359 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.310387 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.312044 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.312555 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.312559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.312560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.312635 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.313037 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.313039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.313040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.313225 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.314255 (+ 1030us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.314347 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.314465 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":126,"threads_started":1}
I20260926 20:15:07.314982 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.302994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.23.50.189:35793)
0926 20:15:07.303460 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.303463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.303476 (+ 13us) server_negotiation.cc:407] Connection header received
0926 20:15:07.303511 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.303516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.303555 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.303622 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.304467 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.305046 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.305786 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.306123 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.306215 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.310558 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.310562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.310565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.310595 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.312422 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.312930 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.312935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.312939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.313021 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.313694 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.313697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.313708 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.314014 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.314322 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.314721 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.314841 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 20:15:07.317436 14194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83: Bootstrap starting.
I20260926 20:15:07.317652 14043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce: Bootstrap starting.
I20260926 20:15:07.318624 23754 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 20:15:07.320307 14043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:07.320554 14194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:07.319747 14017 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b57d84be20c841fab3f3b860725fa5ce"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48172
Sep 26 20:15:07 dist-test-slave-7034 krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790453707, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.50.189@KRBTEST.COM for kudu/127.23.50.188@KRBTEST.COM
I20260926 20:15:07.321180 14043 log.cc:824] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:07.321385 14194 log.cc:824] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:07.323686 14043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce: No bootstrap required, opened a new log
I20260926 20:15:07.323686 14194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83: No bootstrap required, opened a new log
I20260926 20:15:07.325273 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.311745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.23.50.188:38747)
0926 20:15:07.313370 (+ 1625us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.313374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.313385 (+ 11us) server_negotiation.cc:407] Connection header received
0926 20:15:07.313416 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.313419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.313454 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.314120 (+ 666us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.314216 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.315008 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.319010 (+ 4002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.319302 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.319747 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.321710 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.321715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.321717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.321741 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.323502 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.323995 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.323999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.324000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.324094 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.324376 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.324381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.324384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.324632 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.324724 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.325025 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.325133 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1562,"thread_start_us":124,"threads_started":1}
I20260926 20:15:07.325273 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.311986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.50.188:38747 (local address 127.0.0.1:33060)
0926 20:15:07.312171 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 20:15:07.312186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 20:15:07.312278 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 20:15:07.313583 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 20:15:07.313587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 20:15:07.313779 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 20:15:07.313901 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.313908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.318020 (+ 4112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.318024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.318802 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 20:15:07.318813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.319428 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.319432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 20:15:07.319514 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.320006 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 20:15:07.320020 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 20:15:07.321492 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 20:15:07.323608 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.323628 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.323631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.323885 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.324185 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 20:15:07.324189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 20:15:07.324190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 20:15:07.324275 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 20:15:07.324736 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 20:15:07.324741 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 20:15:07.324841 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 20:15:07.324964 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 20:15:07.325108 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 20:15:07.326766 14043 raft_consensus.cc:348] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } }
I20260926 20:15:07.327004 14043 raft_consensus.cc:729] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b57d84be20c841fab3f3b860725fa5ce, State: Initialized, Role: FOLLOWER
I20260926 20:15:07.327348 14043 consensus_queue.cc:260] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } }
I20260926 20:15:07.327651 14194 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } }
I20260926 20:15:07.327874 14194 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5eb4c1f5cae4697bf353613264fdc83, State: Initialized, Role: FOLLOWER
I20260926 20:15:07.327879 14203 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } } }
I20260926 20:15:07.328034 14203 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:07.328191 14194 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } }
I20260926 20:15:07.328663 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } } }
I20260926 20:15:07.328802 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:07.328820 14043 sys_catalog.cc:564] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:15:07.329020 14017 raft_consensus.cc:482] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 20:15:07.329155 14017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:07.329406 14194 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 20:15:07.330339 13996 tablet.cc:2369] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 20:15:07.330524 14017 raft_consensus.cc:504] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } }
I20260926 20:15:07.331388 14017 leader_election.cc:288] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [CANDIDATE]: Term 1 election: Requested vote from peers f08e5d792cc34394ba39241b8a32c02e (127.23.50.189:35793), a5eb4c1f5cae4697bf353613264fdc83 (127.23.50.188:38747)
I20260926 20:15:07.334272 14168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b57d84be20c841fab3f3b860725fa5ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a5eb4c1f5cae4697bf353613264fdc83"
I20260926 20:15:07.334503 14168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [term 0 FOLLOWER]: Advancing to term 1
W20260926 20:15:07.335079 14225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:15:07.335155 14225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:15:07.335667 14168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b57d84be20c841fab3f3b860725fa5ce in term 1.
I20260926 20:15:07.335949 14118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e: Bootstrap starting.
I20260926 20:15:07.336053 13994 leader_election.cc:302] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5eb4c1f5cae4697bf353613264fdc83, b57d84be20c841fab3f3b860725fa5ce; no voters:
I20260926 20:15:07.336560 14203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 20:15:07.336769 14203 raft_consensus.cc:686] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [term 1 LEADER]: Becoming Leader. State: Replica: b57d84be20c841fab3f3b860725fa5ce, State: Running, Role: LEADER
I20260926 20:15:07.337086 14203 consensus_queue.cc:237] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } }
I20260926 20:15:07.338599 14118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e: Neither blocks nor log segments found. Creating new log.
I20260926 20:15:07.339504 14118 log.cc:824] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e: Log is configured to *not* fsync() on all Append() calls
I20260926 20:15:07.337241 13975 master_runner.cc:186] Error getting master registration for 127.23.50.189:35793: OK, instance_id { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" instance_seqno: 1790453707051131 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:15:07.340456 13975 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:15:07.341918 14203 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57d84be20c841fab3f3b860725fa5ce. Latest consensus state: current_term: 1 leader_uuid: "b57d84be20c841fab3f3b860725fa5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } } }
I20260926 20:15:07.342060 14203 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:07.344017 14118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e: No bootstrap required, opened a new log
W20260926 20:15:07.345129 14226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:15:07.345194 14226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:15:07.342635 14125 master_runner.cc:186] Error getting master registration for 127.23.50.189:35793: OK, instance_id { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" instance_seqno: 1790453707051131 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 20:15:07.345953 14125 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 20:15:07.348193 14118 raft_consensus.cc:348] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } }
I20260926 20:15:07.348414 14118 raft_consensus.cc:729] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08e5d792cc34394ba39241b8a32c02e, State: Initialized, Role: FOLLOWER
I20260926 20:15:07.348749 14118 consensus_queue.cc:260] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } }
I20260926 20:15:07.350487 14118 sys_catalog.cc:564] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 20:15:07.350795 14092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b57d84be20c841fab3f3b860725fa5ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f08e5d792cc34394ba39241b8a32c02e"
I20260926 20:15:07.351029 14092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [term 0 FOLLOWER]: Advancing to term 1
I20260926 20:15:07.352573 14092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b57d84be20c841fab3f3b860725fa5ce in term 1.
I20260926 20:15:07.352734 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.336376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.23.50.189:35793)
0926 20:15:07.336493 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.336499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.341409 (+ 4910us) server_negotiation.cc:407] Connection header received
0926 20:15:07.341573 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.341577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.341626 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.341678 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.343624 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.344429 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.345707 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.346062 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.346146 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.348096 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.348100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.348102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.348130 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.350127 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.351272 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.351277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.351278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.351402 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.351750 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.351754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.351755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.351971 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.352069 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.352463 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.352586 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:15:07.353020 14240 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } } }
I20260926 20:15:07.353144 14240 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:07.354923 14238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 20:15:07.356007 14238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 20:15:07.360419 14251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 20:15:07.360504 14251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 20:15:07.365733 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.338462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.23.50.188:38747)
0926 20:15:07.338581 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.338586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.345492 (+ 6906us) server_negotiation.cc:407] Connection header received
0926 20:15:07.345659 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.345662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.345706 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.345755 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.357959 (+ 12204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.358641 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.359727 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.360037 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.360111 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.362054 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.362058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.362061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.362089 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.363711 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.364171 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.364175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.364177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.364258 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.364529 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.364532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.364534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.364743 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.364862 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.365452 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.365564 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 20:15:07.369016 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:07.341081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.23.50.190:43967)
0926 20:15:07.341183 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.341187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.346821 (+ 5634us) server_negotiation.cc:407] Connection header received
0926 20:15:07.347018 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.347024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.347077 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.347149 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.349948 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.350764 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.354695 (+ 3931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:07.355015 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:07.355113 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:07.363956 (+ 8843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 20:15:07.363960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 20:15:07.363963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 20:15:07.363997 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 20:15:07.365681 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.367613 (+ 1932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.367616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.367618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.367707 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 20:15:07.368025 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 20:15:07.368028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 20:15:07.368030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 20:15:07.368272 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 20:15:07.368395 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:07.368738 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 20:15:07.368859 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 20:15:07.371028 14168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [term 1 FOLLOWER]: Refusing update from remote peer b57d84be20c841fab3f3b860725fa5ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:07.371127 14092 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [term 1 FOLLOWER]: Refusing update from remote peer b57d84be20c841fab3f3b860725fa5ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 20:15:07.371898 14203 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:07.372123 14232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 20:15:07.375931 14240 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57d84be20c841fab3f3b860725fa5ce. Latest consensus state: current_term: 1 leader_uuid: "b57d84be20c841fab3f3b860725fa5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } } }
I20260926 20:15:07.376061 14240 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:07.377348 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b57d84be20c841fab3f3b860725fa5ce. Latest consensus state: current_term: 1 leader_uuid: "b57d84be20c841fab3f3b860725fa5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } } }
I20260926 20:15:07.377508 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:07.386365 14203 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b57d84be20c841fab3f3b860725fa5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } } }
I20260926 20:15:07.386488 14203 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:07.386825 14253 mvcc.cc:205] Tried to move back new op lower bound from 7333698385384189952 to 7333698385261961216. Current Snapshot: MvccSnapshot[applied={T|T < 7333698385384189952}]
I20260926 20:15:07.386837 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b57d84be20c841fab3f3b860725fa5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } } }
I20260926 20:15:07.386958 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:07.387112 14256 mvcc.cc:205] Tried to move back new op lower bound from 7333698385384189952 to 7333698385261961216. Current Snapshot: MvccSnapshot[applied={T|T < 7333698385384189952}]
I20260926 20:15:07.390479 14232 sys_catalog.cc:455] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b57d84be20c841fab3f3b860725fa5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } } }
I20260926 20:15:07.390592 14232 sys_catalog.cc:458] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce [sys.catalog]: This master's current role is: LEADER
I20260926 20:15:07.392089 14240 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b57d84be20c841fab3f3b860725fa5ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b57d84be20c841fab3f3b860725fa5ce" member_type: VOTER last_known_addr { host: "127.23.50.190" port: 43967 } } peers { permanent_uuid: "f08e5d792cc34394ba39241b8a32c02e" member_type: VOTER last_known_addr { host: "127.23.50.189" port: 35793 } } peers { permanent_uuid: "a5eb4c1f5cae4697bf353613264fdc83" member_type: VOTER last_known_addr { host: "127.23.50.188" port: 38747 } } }
I20260926 20:15:07.392194 14240 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 20:15:07.393241 14254 mvcc.cc:205] Tried to move back new op lower bound from 7333698385384189952 to 7333698385261961216. Current Snapshot: MvccSnapshot[applied={T|T < 7333698385384189952}]
I20260926 20:15:07.397966 14238 catalog_manager.cc:1347] Generated new cluster ID: 8cc1d12d1dd44812b6ff8566b1e38909
I20260926 20:15:07.398036 14238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 20:15:07.425891 14238 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 20:15:07.427169 14238 catalog_manager.cc:1504] Loading token signing keys...
I20260926 20:15:07.444254 14238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b57d84be20c841fab3f3b860725fa5ce: Generated new TSK 0
I20260926 20:15:07.447561 14238 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 20:15:07.486735 14124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:07.483221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33450 (local address 127.23.50.189:35793)
0926 20:15:07.483404 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.483409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.485378 (+ 1969us) server_negotiation.cc:407] Connection header received
0926 20:15:07.485792 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.485796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.485874 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.485932 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.486601 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33450: BlockingRecv error: recv got EOF from 127.0.0.1:33450 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260926 20:15:07.488142 14197 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:07.484034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.23.50.188:38747)
0926 20:15:07.484148 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.484153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.485707 (+ 1554us) server_negotiation.cc:407] Connection header received
0926 20:15:07.487337 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.487341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.487392 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.487444 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.487998 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33076: BlockingRecv error: recv got EOF from 127.0.0.1:33076 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 20:15:07.489253 14046 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:07.484731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.23.50.190:43967)
0926 20:15:07.484838 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.484842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.487135 (+ 2293us) server_negotiation.cc:407] Connection header received
0926 20:15:07.488503 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.488506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.488557 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.488610 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.489138 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48188: BlockingRecv error: recv got EOF from 127.0.0.1:48188 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 20:15:07.494125 14124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:07.492565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.23.50.189:35793)
0926 20:15:07.492667 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.492672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.493098 (+ 426us) server_negotiation.cc:407] Connection header received
0926 20:15:07.493330 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.493334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.493408 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.493462 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.494022 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33460: BlockingRecv error: recv got EOF from 127.0.0.1:33460 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 20:15:07.499234 14124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:07.497657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.23.50.189:35793)
0926 20:15:07.497768 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.497773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.498351 (+ 578us) server_negotiation.cc:407] Connection header received
0926 20:15:07.498543 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.498546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.498596 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.498647 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.499140 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33470: BlockingRecv error: recv got EOF from 127.0.0.1:33470 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 20:15:07.504911 14124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:07.503337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.23.50.189:35793)
0926 20:15:07.503445 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.503451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.503966 (+ 515us) server_negotiation.cc:407] Connection header received
0926 20:15:07.504157 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.504161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.504210 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.504260 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.504803 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33474: BlockingRecv error: recv got EOF from 127.0.0.1:33474 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 20:15:07.512775 14124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:07.510972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.23.50.189:35793)
0926 20:15:07.511107 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.511111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.511843 (+ 732us) server_negotiation.cc:407] Connection header received
0926 20:15:07.512044 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.512047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.512099 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.512152 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.512674 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33476: BlockingRecv error: recv got EOF from 127.0.0.1:33476 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 20:15:07.525081 14124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:07.523126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.23.50.189:35793)
0926 20:15:07.523252 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.523257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.523900 (+ 643us) server_negotiation.cc:407] Connection header received
0926 20:15:07.524137 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.524141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.524200 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.524258 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.524939 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33486: BlockingRecv error: recv got EOF from 127.0.0.1:33486 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 20:15:07.545822 14124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:07.543817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.23.50.189:35793)
0926 20:15:07.543944 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.543949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.544679 (+ 730us) server_negotiation.cc:407] Connection header received
0926 20:15:07.544894 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.544898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.544953 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.545008 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.545659 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33492: BlockingRecv error: recv got EOF from 127.0.0.1:33492 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 20:15:07.581470 14124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:07.580253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.23.50.189:35793)
0926 20:15:07.580387 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.580392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.580535 (+ 143us) server_negotiation.cc:407] Connection header received
0926 20:15:07.580665 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.580669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.580729 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.580834 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.581355 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33502: BlockingRecv error: recv got EOF from 127.0.0.1:33502 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 20:15:07.648766 14124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:07.647660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.23.50.189:35793)
0926 20:15:07.647816 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.647820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.647952 (+ 132us) server_negotiation.cc:407] Connection header received
0926 20:15:07.648092 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.648095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.648144 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.648230 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.648646 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33504: BlockingRecv error: recv got EOF from 127.0.0.1:33504 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 20:15:07.780174 14124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:07.779082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.23.50.189:35793)
0926 20:15:07.779258 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:07.779261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:07.779278 (+ 17us) server_negotiation.cc:407] Connection header received
0926 20:15:07.779424 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:07.779427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:07.779476 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:07.779547 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:07.780014 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33506: BlockingRecv error: recv got EOF from 127.0.0.1:33506 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 20:15:08.039455 14124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 20:15:08.038329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.23.50.189:35793)
0926 20:15:08.038480 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:08.038484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:08.038581 (+ 97us) server_negotiation.cc:407] Connection header received
0926 20:15:08.038721 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:08.038723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:08.038773 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:08.038857 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 20:15:08.039352 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33510: BlockingRecv error: recv got EOF from 127.0.0.1:33510 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 20:15:08.346570 14226 catalog_manager.cc:1259] Loaded cluster ID: 8cc1d12d1dd44812b6ff8566b1e38909
I20260926 20:15:08.346657 14226 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83: loading cluster ID for follower catalog manager: success
I20260926 20:15:08.348416 14226 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83: acquiring CA information for follower catalog manager: success
I20260926 20:15:08.349591 14226 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5eb4c1f5cae4697bf353613264fdc83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:15:08.361752 14251 catalog_manager.cc:1259] Loaded cluster ID: 8cc1d12d1dd44812b6ff8566b1e38909
I20260926 20:15:08.361829 14251 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e: loading cluster ID for follower catalog manager: success
I20260926 20:15:08.363533 14251 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e: acquiring CA information for follower catalog manager: success
I20260926 20:15:08.364660 14251 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f08e5d792cc34394ba39241b8a32c02e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 20:15:08.557528 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:08.553665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.23.50.189:35793)
0926 20:15:08.553960 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:08.553965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:08.554011 (+ 46us) server_negotiation.cc:407] Connection header received
0926 20:15:08.554160 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:08.554163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:08.554311 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:08.554421 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:08.554660 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:08.555178 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:08.556082 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:08.556320 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:08.556402 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:08.556728 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:08.557177 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:08.557272 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:08.557349 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 20:15:08.557400 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":103,"threads_started":1}
I20260926 20:15:08.564265 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:08.560562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.23.50.188:38747)
0926 20:15:08.560801 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:08.560804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:08.560818 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:15:08.561009 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:08.561012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:08.561150 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:08.561258 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:08.561480 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:08.562084 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:08.562904 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:08.563142 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:08.563226 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:08.563502 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:08.563935 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:08.564023 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:08.564092 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 20:15:08.564139 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260926 20:15:08.571350 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:08.567685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.23.50.188:38747)
0926 20:15:08.567809 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:08.567813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:08.568041 (+ 228us) server_negotiation.cc:407] Connection header received
0926 20:15:08.568235 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:08.568238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:08.568313 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:08.568386 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:08.568617 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:08.569313 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:08.570323 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:08.570544 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:08.570656 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:08.570906 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:08.570974 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:08.571055 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:08.571142 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 20:15:08.571193 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 20:15:08.571724 14325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:08.567141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33538 (local address 127.23.50.189:35793)
0926 20:15:08.567294 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:08.567299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:08.567807 (+ 508us) server_negotiation.cc:407] Connection header received
0926 20:15:08.567871 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:08.567874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:08.567932 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:08.568021 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:08.568320 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:08.569108 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:08.570310 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:08.570653 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:08.570742 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:08.571243 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:08.571322 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:08.571368 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:08.571524 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 20:15:08.571578 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 20:15:08.572274 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 20:15:08.567271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.23.50.190:43967)
0926 20:15:08.567540 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 20:15:08.567544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 20:15:08.567558 (+ 14us) server_negotiation.cc:407] Connection header received
0926 20:15:08.567744 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 20:15:08.567751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 20:15:08.567920 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 20:15:08.568054 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 20:15:08.568404 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:08.569002 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:08.570323 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 20:15:08.570653 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 20:15:08.570742 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 20:15:08.571295 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 20:15:08.571690 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 20:15:08.571782 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 20:15:08.572051 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 20:15:08.572126 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260926 20:15:08.575533 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 13975
I20260926 20:15:08.582054 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 14050
I20260926 20:15:08.588716 23754 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFqzGjO/build/debug/bin/kudu with pid 14125
2026-09-26T20:15:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3126 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6291 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (287356 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST